Analysis with gcov while running the stress-ng urandom stressor
shows that there are a couple of fortify panic paths that are highly
unlikely to be executed for well-behaving code. Adding appropriate
branch hints improves the stress-ng urandom stressor my a small
but statistically measureable amount. Ran 100 x 1 minute tests and
measured the stressor bogo-op rates on a Debian based Intel(R)
Core(TM) Ultra 9 285K with a 6.15 kernel with turbo disabled to
reduce jitter.
Results based on a Geometic Mean of 100 tests:
Without patch: 50512.95 bogo-ops/sec
With patch: 50819.58 bogo-ops/sec
%Std.Deviation of ~0.18%, so low jitter in results, improvement of ~0.6%
Branch hints can only be enabled if CONFIG_TRACE_BRANCH_PROFILING due
to a static variable being declared in __branch_check__ when using
trace branch profiling in the unlikely macro causing build issues.
Signed-off-by: Colin Ian King <colin.i.king@xxxxxxxxx>
---
V2: add fortify_unlikely macro wrapper for CONFIG_TRACE_BRANCH_PROFILING
build config.
---
include/linux/fortify-string.h | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/include/linux/fortify-string.h b/include/linux/fortify-string.h
index e4ce1cae03bf..c740114bcbf8 100644
--- a/include/linux/fortify-string.h
+++ b/include/linux/fortify-string.h
@@ -10,6 +10,12 @@
#define __FORTIFY_INLINE extern __always_inline __gnu_inline __overloadable
#define __RENAME(x) __asm__(#x)
+#if defined(CONFIG_TRACE_BRANCH_PROFILING)
+#define fortify_unlikely(x) (x)
+#else
+#define fortify_unlikely(x) unlikely(x)
+#endif
+
#define FORTIFY_REASON_DIR(r) FIELD_GET(BIT(0), r)
#define FORTIFY_REASON_FUNC(r) FIELD_GET(GENMASK(7, 1), r)
#define FORTIFY_REASON(func, write) (FIELD_PREP(BIT(0), write) | \
@@ -593,9 +599,9 @@ __FORTIFY_INLINE bool fortify_memcpy_chk(__kernel_size_t size,
* (The SIZE_MAX test is to optimize away checks where the buffer
* lengths are unknown.)
*/
- if (p_size != SIZE_MAX && p_size < size)
+ if (fortify_unlikely(p_size != SIZE_MAX && p_size < size))
fortify_panic(func, FORTIFY_WRITE, p_size, size, true);
- else if (q_size != SIZE_MAX && q_size < size)
+ else if (fortify_unlikely(q_size != SIZE_MAX && q_size < size))
fortify_panic(func, FORTIFY_READ, p_size, size, true);
/*
--
2.50.0
