Hello, for this commit, we reported "[brauner-vfs:vfs-6.14.misc] [pipe_read] aaec5a95d5: hackbench.throughput 7.5% regression" in https://lore.kernel.org/all/202501101015.90874b3a-lkp@xxxxxxxxx/ and "[linux-next:master] [pipe_read] aaec5a95d5: stress-ng.poll.ops_per_sec 11.1% regression" in https://lore.kernel.org/all/202501201311.6d25a0b9-lkp@xxxxxxxxx/ we also noticed in https://lore.kernel.org/all/20250123125607.GA16498@xxxxxxxxxx/ that the hackbench regression is reproducible. now we noticed the commit is in mainline and still found similar regression. (as well as an improvement as below details) we don't have enough background knowledge that if the benefit of this change is more important than some small regression on micro benchmark, this report is just FYI what we observed in our tests. kernel test robot noticed a 6.4% regression of hackbench.throughput on: commit: aaec5a95d59615523db03dd53c2052f0a87beea7 ("pipe_read: don't wake up the writer if the pipe is still full") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master [still regression on linus/master 78f4e737a53e1163ded2687a922fce138aee73f5] [still regression on linux-next/master 2ae2aaafb21454f4781c30734959cf223ab486ef] testcase: hackbench config: x86_64-rhel-9.4 compiler: gcc-12 test machine: 4 threads Intel(R) Xeon(R) CPU E3-1225 v5 @ 3.30GHz (Skylake) with 16G memory parameters: nr_threads: 50% iterations: 4 mode: process ipc: pipe cpufreq_governor: performance In addition to that, the commit also has significant impact on the following tests: +------------------+---------------------------------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput 7.7% regression | | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | iterations=4 | | | mode=threads | | | nr_threads=50% | +------------------+---------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.poll.ops_per_sec 11.1% regression | | test machine | 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_threads=100% | | | test=poll | | | testtime=60s | +------------------+---------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.tee.ops_per_sec 500.7% improvement | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | cpufreq_governor=performance | | | nr_threads=100% | | | test=tee | | | testtime=60s | +------------------+---------------------------------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput 7.5% regression | | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | iterations=4 | | | mode=process | | | nr_threads=50% | +------------------+---------------------------------------------------------------------------------------------+ If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot <oliver.sang@xxxxxxxxx> | Closes: https://lore.kernel.org/oe-lkp/202506261329.8c74199e-lkp@xxxxxxxxx Details are as below: --------------------------------------------------------------------------------------------------> The kernel config and materials to reproduce are available at: https://download.01.org/0day-ci/archive/20250626/202506261329.8c74199e-lkp@xxxxxxxxx ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-12/performance/pipe/4/x86_64-rhel-9.4/process/50%/debian-12-x86_64-20240206.cgz/lkp-skl-d06/hackbench commit: d2fc0ed52a ("Merge branch 'vfs-6.14.uncached_buffered_io'") aaec5a95d5 ("pipe_read: don't wake up the writer if the pipe is still full") d2fc0ed52a284a13 aaec5a95d59615523db03dd53c2 ---------------- --------------------------- %stddev %change %stddev \ | \ 3255 ± 2% +11.7% 3637 perf-c2c.HITM.local 339169 ± 2% +15.4% 391366 vmstat.system.cs 71801 ± 2% +38.8% 99667 ± 2% vmstat.system.in 32837 ± 3% -6.4% 30733 ± 2% proc-vmstat.nr_mapped 3900380 -8.4% 3573143 proc-vmstat.numa_hit 3901829 -8.4% 3573143 proc-vmstat.numa_local 3931376 -8.2% 3607706 proc-vmstat.pgalloc_normal 3715390 -8.5% 3398913 proc-vmstat.pgfree 27292 -6.4% 25535 hackbench.throughput 26387 -4.8% 25121 hackbench.throughput_avg 27292 -6.4% 25535 hackbench.throughput_best 5171648 ± 3% +28.2% 6632375 ± 2% hackbench.time.involuntary_context_switches 154.57 +7.8% 166.62 hackbench.time.system_time 18478209 ± 2% +19.8% 22145567 hackbench.time.voluntary_context_switches 16.66 ±100% -10.7 5.98 ±205% perf-profile.calltrace.cycles-pp.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 16.66 ±100% -10.7 5.98 ±205% perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter.vfs_write 16.66 ±100% -10.7 5.98 ±205% perf-profile.calltrace.cycles-pp.shmem_write_begin.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 16.66 ±100% -10.7 5.98 ±205% perf-profile.children.cycles-pp.shmem_alloc_and_add_folio 16.66 ±100% -10.7 5.98 ±205% perf-profile.children.cycles-pp.shmem_get_folio_gfp 16.66 ±100% -10.7 5.98 ±205% perf-profile.children.cycles-pp.shmem_write_begin 9402 ± 8% -16.5% 7847 ± 2% sched_debug.cfs_rq:/.runnable_avg.avg 1045 ± 22% -24.3% 791.38 ± 11% sched_debug.cfs_rq:/.util_est.avg 610.42 ± 45% -57.0% 262.42 ± 66% sched_debug.cfs_rq:/.util_est.min 2616103 ± 2% +14.4% 2991704 sched_debug.cpu.nr_switches.avg 2689656 +14.9% 3089911 sched_debug.cpu.nr_switches.max 2561049 ± 2% +12.5% 2882129 ± 3% sched_debug.cpu.nr_switches.min 0.84 +9.0% 0.92 perf-stat.i.MPKI 1.99 +0.1 2.05 perf-stat.i.branch-miss-rate% 34496253 +3.7% 35766259 perf-stat.i.branch-misses 2.63 +0.1 2.75 perf-stat.i.cache-miss-rate% 7514001 +8.9% 8180511 perf-stat.i.cache-misses 2.998e+08 +5.0% 3.149e+08 perf-stat.i.cache-references 347430 ± 2% +15.6% 401508 perf-stat.i.context-switches 3115 -67.6% 1010 ± 5% perf-stat.i.cpu-migrations 2213 -11.2% 1966 perf-stat.i.cycles-between-cache-misses 87.19 ± 2% +15.3% 100.54 perf-stat.i.metric.K/sec 1688 -2.0% 1655 perf-stat.i.minor-faults 1688 -2.0% 1655 perf-stat.i.page-faults 0.87 +8.4% 0.94 perf-stat.overall.MPKI 1.98 +0.1 2.05 perf-stat.overall.branch-miss-rate% 2.51 +0.1 2.60 perf-stat.overall.cache-miss-rate% 1797 -8.1% 1652 perf-stat.overall.cycles-between-cache-misses 34012373 +3.7% 35283212 perf-stat.ps.branch-misses 7409882 +8.9% 8071103 perf-stat.ps.cache-misses 2.956e+08 +5.1% 3.107e+08 perf-stat.ps.cache-references 342572 ± 2% +15.6% 396160 perf-stat.ps.context-switches 3071 -67.5% 996.99 ± 5% perf-stat.ps.cpu-migrations 1665 -1.9% 1633 perf-stat.ps.minor-faults 1665 -1.9% 1633 perf-stat.ps.page-faults 6.128e+11 +5.5% 6.463e+11 perf-stat.total.instructions 0.14 ± 10% -16.0% 0.12 ± 5% perf-sched.sch_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.31 ± 4% -15.6% 0.26 ± 5% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.55 ± 6% +52.0% 0.83 ± 6% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 2.88 ± 13% +105.9% 5.93 ± 18% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 3.29 ± 5% +126.6% 7.46 ± 7% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 0.14 ± 8% +27.1% 0.17 ± 13% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.69 ± 6% +713.7% 5.62 perf-sched.sch_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 0.66 ± 24% +75.5% 1.16 ± 20% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.25 ± 6% +61.3% 0.40 ± 4% perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 34.33 ± 10% +25.0% 42.92 ± 10% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 33.35 ± 21% +40.9% 47.00 ± 15% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 0.66 ± 4% -16.4% 0.55 ± 5% perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 1.33 ± 6% +85.1% 2.45 ± 6% perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 6.57 ± 36% +80.8% 11.88 ± 18% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 8.78 ± 7% +173.3% 24.00 ± 5% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 0.28 ± 10% +32.7% 0.37 ± 15% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 2.16 ± 5% +768.0% 18.79 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 11.16 ± 18% +69.1% 18.88 ± 15% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 5.41 ± 11% +38.3% 7.48 ± 14% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.58 ± 7% +98.6% 1.16 ± 4% perf-sched.wait_and_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 5243 ± 5% +36.9% 7176 ± 6% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 4198 ± 3% +72.3% 7231 ± 2% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 520.33 ± 4% +71.8% 893.83 ± 2% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 25991 ± 9% -29.5% 18330 ± 12% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 8647 ± 5% +51.1% 13067 ± 3% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown] 98274 ± 5% -93.4% 6480 ± 2% perf-sched.wait_and_delay.count.pipe_write.vfs_write.ksys_write.do_syscall_64 98.00 ± 34% +74.5% 171.00 ± 15% perf-sched.wait_and_delay.count.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 139344 ± 6% -27.5% 101087 ± 4% perf-sched.wait_and_delay.count.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 71.46 ± 14% +38.1% 98.71 ± 12% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 673.17 ± 13% +16.9% 787.17 ± 4% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.35 ± 4% -17.1% 0.29 ± 5% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.78 ± 6% +108.2% 1.62 ± 6% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 5.49 ± 8% +201.3% 16.55 ± 4% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 0.14 ± 12% +38.5% 0.20 ± 20% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 1.47 ± 4% +793.5% 13.17 ± 2% perf-sched.wait_time.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 6.76 ± 14% +99.6% 13.50 ± 16% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 4.75 ± 15% +33.2% 6.33 ± 14% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 35.39 ± 11% -19.9% 28.35 ± 9% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.34 ± 8% +126.1% 0.76 ± 4% perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 63.41 ± 25% +72.9% 109.66 ± 14% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 45.83 ± 18% +56.6% 71.79 ± 17% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 673.16 ± 13% +16.9% 787.17 ± 4% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread *************************************************************************************************** lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-12/performance/pipe/4/x86_64-rhel-9.4/threads/50%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp2/hackbench commit: d2fc0ed52a ("Merge branch 'vfs-6.14.uncached_buffered_io'") aaec5a95d5 ("pipe_read: don't wake up the writer if the pipe is still full") d2fc0ed52a284a13 aaec5a95d59615523db03dd53c2 ---------------- --------------------------- %stddev %change %stddev \ | \ 5.042e+08 ± 17% -18.7% 4.1e+08 ± 2% cpuidle..time 10652969 ± 6% -28.3% 7635249 ± 6% cpuidle..usage 4.47 ± 21% -1.4 3.07 ± 6% mpstat.cpu.all.idle% 0.03 ± 4% -0.0 0.03 mpstat.cpu.all.soft% 1452 -14.4% 1243 vmstat.procs.r 1402445 +25.6% 1761120 vmstat.system.in 2894369 ± 22% -34.7% 1889346 ± 12% numa-meminfo.node1.Active 2894369 ± 22% -34.7% 1889346 ± 12% numa-meminfo.node1.Active(anon) 2473459 ± 33% -42.9% 1412836 ± 22% numa-meminfo.node1.Shmem 724297 ± 22% -34.7% 472642 ± 12% numa-vmstat.node1.nr_active_anon 618989 ± 33% -42.9% 353537 ± 21% numa-vmstat.node1.nr_shmem 724296 ± 22% -34.7% 472642 ± 12% numa-vmstat.node1.nr_zone_active_anon 98180 ± 5% +59.7% 156837 ± 4% perf-c2c.HITM.local 3639 ± 11% +82.8% 6651 ± 21% perf-c2c.HITM.remote 101819 ± 5% +60.6% 163489 ± 4% perf-c2c.HITM.total 3267360 ± 25% -32.4% 2208241 ± 14% meminfo.Active 3267360 ± 25% -32.4% 2208241 ± 14% meminfo.Active(anon) 5982182 ± 13% -17.5% 4933057 ± 6% meminfo.Cached 3425356 ± 24% -30.9% 2366882 ± 13% meminfo.Committed_AS 9822130 ± 8% -12.0% 8641150 ± 4% meminfo.Memused 2479849 ± 33% -42.3% 1430723 ± 22% meminfo.Shmem 9900363 ± 8% -11.4% 8768940 ± 3% meminfo.max_used_kB 823004 -7.7% 759380 hackbench.throughput 788354 -7.0% 732864 hackbench.throughput_avg 823004 -7.7% 759380 hackbench.throughput_best 714741 -5.7% 674114 hackbench.throughput_worst 76.78 +7.5% 82.53 hackbench.time.elapsed_time 76.78 +7.5% 82.53 hackbench.time.elapsed_time.max 1.971e+08 +30.4% 2.57e+08 hackbench.time.involuntary_context_switches 52109 ± 3% +25.6% 65473 ± 7% hackbench.time.minor_page_faults 8193 +9.5% 8974 hackbench.time.system_time 1209 +2.4% 1238 hackbench.time.user_time 813559 ± 25% -32.1% 552002 ± 14% proc-vmstat.nr_active_anon 197770 -1.1% 195674 proc-vmstat.nr_anon_pages 1492419 ± 13% -17.4% 1233047 ± 6% proc-vmstat.nr_file_pages 616835 ± 33% -42.0% 357462 ± 22% proc-vmstat.nr_shmem 30639 -1.9% 30072 proc-vmstat.nr_slab_reclaimable 813559 ± 25% -32.1% 552002 ± 14% proc-vmstat.nr_zone_active_anon 232915 ± 8% -19.5% 187586 ± 15% proc-vmstat.numa_hint_faults 1.131e+08 -2.3% 1.105e+08 proc-vmstat.numa_hit 1.13e+08 -2.3% 1.104e+08 proc-vmstat.numa_local 1.132e+08 -2.3% 1.106e+08 proc-vmstat.pgalloc_normal 1.117e+08 -1.8% 1.097e+08 proc-vmstat.pgfree 5.855e+10 -5.8% 5.513e+10 perf-stat.i.branch-instructions 0.45 +0.0 0.47 perf-stat.i.branch-miss-rate% 2.443e+08 -1.8% 2.399e+08 perf-stat.i.branch-misses 13.43 ± 3% -4.3 9.18 ± 6% perf-stat.i.cache-miss-rate% 1.239e+09 ± 4% +39.5% 1.729e+09 perf-stat.i.cache-references 1.26 +6.9% 1.35 perf-stat.i.cpi 577288 ± 2% +53.7% 887271 perf-stat.i.cpu-migrations 2.522e+11 -5.7% 2.378e+11 perf-stat.i.instructions 0.80 -6.3% 0.75 perf-stat.i.ipc 98.67 +2.5% 101.14 perf-stat.i.metric.K/sec 9540 ± 3% -9.7% 8612 ± 4% perf-stat.i.minor-faults 9540 ± 3% -9.7% 8612 ± 4% perf-stat.i.page-faults 0.35 ± 44% +0.1 0.43 perf-stat.overall.branch-miss-rate% 1.05 ± 44% +28.3% 1.35 perf-stat.overall.cpi 1.019e+09 ± 44% +67.4% 1.705e+09 perf-stat.ps.cache-references 470800 ± 44% +85.7% 874307 perf-stat.ps.cpu-migrations 1.611e+13 ± 44% +22.0% 1.966e+13 perf-stat.total.instructions 6.14 ± 4% -12.7% 5.36 ± 5% sched_debug.cfs_rq:/.h_nr_running.avg 18.00 ± 14% -28.3% 12.90 ± 13% sched_debug.cfs_rq:/.h_nr_running.max 3.94 ± 23% -29.5% 2.78 sched_debug.cfs_rq:/.h_nr_running.stddev 6216 ± 2% -11.3% 5516 ± 2% sched_debug.cfs_rq:/.runnable_avg.avg 13241 ± 13% -22.6% 10251 ± 10% sched_debug.cfs_rq:/.runnable_avg.max 2459 ± 38% -38.6% 1510 ± 8% sched_debug.cfs_rq:/.runnable_avg.stddev 278.40 ± 7% -8.8% 253.92 ± 3% sched_debug.cfs_rq:/.util_avg.stddev 6923 ± 3% -16.5% 5782 sched_debug.cpu.curr->pid.avg 9278 -13.6% 8017 sched_debug.cpu.curr->pid.max 1850 ± 22% -45.1% 1015 ± 13% sched_debug.cpu.curr->pid.stddev 7640 ± 28% -72.3% 2117 ±128% sched_debug.cpu.max_idle_balance_cost.stddev 6.10 ± 4% -12.0% 5.37 ± 5% sched_debug.cpu.nr_running.avg 18.00 ± 14% -28.3% 12.90 ± 13% sched_debug.cpu.nr_running.max 3.96 ± 23% -29.9% 2.78 ± 2% sched_debug.cpu.nr_running.stddev 3401286 ± 11% -12.9% 2963839 sched_debug.cpu.nr_switches.max 2211723 ± 7% +17.1% 2589948 sched_debug.cpu.nr_switches.min 148894 ± 18% -54.6% 67575 ± 16% sched_debug.cpu.nr_switches.stddev 14.58 ± 40% -11.7 2.86 ±199% perf-profile.calltrace.cycles-pp.__cmd_record 11.53 ± 72% -11.5 0.00 perf-profile.calltrace.cycles-pp.__ordered_events__flush.perf_session__process_user_event.reader__read_event.perf_session__process_events.record__finish_output 11.53 ± 72% -11.5 0.00 perf-profile.calltrace.cycles-pp.perf_session__process_user_event.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record 14.22 ± 46% -11.4 2.86 ±199% perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.__cmd_record 14.22 ± 46% -11.4 2.86 ±199% perf-profile.calltrace.cycles-pp.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record 14.22 ± 46% -11.4 2.86 ±199% perf-profile.calltrace.cycles-pp.record__finish_output.__cmd_record 9.34 ± 71% -9.3 0.00 perf-profile.calltrace.cycles-pp.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event.perf_session__process_events 6.13 ± 71% -6.1 0.00 perf-profile.calltrace.cycles-pp.evlist__parse_sample.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event.reader__read_event 4.23 ± 78% -4.2 0.00 perf-profile.calltrace.cycles-pp.evsel__parse_sample.evlist__parse_sample.perf_session__deliver_event.__ordered_events__flush.perf_session__process_user_event 24.02 ± 18% -11.6 12.41 ± 40% perf-profile.children.cycles-pp.__cmd_record 11.53 ± 72% -11.5 0.00 perf-profile.children.cycles-pp.__ordered_events__flush 11.53 ± 72% -11.5 0.00 perf-profile.children.cycles-pp.perf_session__process_user_event 14.22 ± 46% -11.4 2.86 ±199% perf-profile.children.cycles-pp.perf_session__process_events 14.22 ± 46% -11.4 2.86 ±199% perf-profile.children.cycles-pp.reader__read_event 14.22 ± 46% -11.4 2.86 ±199% perf-profile.children.cycles-pp.record__finish_output 9.62 ± 72% -9.6 0.00 perf-profile.children.cycles-pp.perf_session__deliver_event 6.13 ± 71% -6.1 0.00 perf-profile.children.cycles-pp.evlist__parse_sample 4.23 ± 78% -4.2 0.00 perf-profile.children.cycles-pp.evsel__parse_sample 1.14 ± 73% +2.4 3.57 ± 32% perf-profile.children.cycles-pp._raw_spin_lock 4.23 ± 78% -4.2 0.00 perf-profile.self.cycles-pp.evsel__parse_sample 11.45 ± 31% +96.2% 22.47 ± 8% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.71 ± 89% -84.9% 0.11 ± 96% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region 0.17 ±127% -97.5% 0.00 ±123% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 5.99 ± 31% +275.3% 22.48 ± 12% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 0.97 ±110% +3354.5% 33.55 ±162% perf-sched.sch_delay.avg.ms.__cond_resched.shmem_inode_acct_blocks.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin 14.75 ± 34% +431.2% 78.36 ± 26% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 1.27 ± 51% +499.2% 7.58 ± 35% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 1.12 ± 30% +60.4% 1.79 ± 12% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 6.39 ± 35% +843.3% 60.32 ± 14% perf-sched.sch_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 17.90 ± 58% +176.4% 49.48 ± 26% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 14.40 ± 45% +295.1% 56.90 ± 84% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1.40 ± 27% +274.3% 5.24 ± 11% perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 510.34 ± 65% -92.2% 39.72 ± 47% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.shmem_alloc_folio 32.40 ±170% -93.0% 2.26 ± 66% perf-sched.sch_delay.max.ms.__cond_resched.__mutex_lock.constprop.0.pipe_read 0.64 ±166% -99.3% 0.00 ±123% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 555.37 ± 23% +49.0% 827.50 ± 15% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 943.44 ± 41% +197.2% 2803 ± 23% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 810.08 ± 36% +171.4% 2198 ± 16% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 192.58 ± 77% -95.5% 8.67 ±148% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown] 543.49 ± 54% +103.5% 1105 ± 40% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 1677 ± 19% +133.6% 3917 ± 15% perf-sched.sch_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 855.77 ± 35% +159.6% 2221 ± 27% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 1552 ± 25% +161.6% 4061 ± 17% perf-sched.sch_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.64 ± 30% +74.2% 2.86 ± 12% perf-sched.total_sch_delay.average.ms 5.08 ± 30% +64.1% 8.33 ± 12% perf-sched.total_wait_and_delay.average.ms 4913449 ± 35% -47.7% 2568016 ± 23% perf-sched.total_wait_and_delay.count.ms 3.44 ± 31% +59.3% 5.47 ± 12% perf-sched.total_wait_time.average.ms 27.82 ± 25% +61.5% 44.93 ± 8% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 18.33 ± 33% +256.8% 65.40 ± 11% perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 45.92 ± 35% +409.9% 234.14 ± 19% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 3.25 ± 31% +56.1% 5.07 ± 12% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 20.64 ± 35% +757.2% 176.94 ± 14% perf-sched.wait_and_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 53.51 ± 55% +168.5% 143.67 ± 22% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 51.14 ± 42% +182.7% 144.54 ± 61% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 4.14 ± 30% +268.0% 15.24 ± 12% perf-sched.wait_and_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 61.67 ± 20% -64.3% 22.00 ± 35% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown] 3724906 ± 34% -42.3% 2149531 ± 23% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 444284 ± 37% -95.4% 20647 ± 25% perf-sched.wait_and_delay.count.pipe_write.vfs_write.ksys_write.do_syscall_64 16.83 ± 23% -65.5% 5.80 ± 63% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 172.17 ± 55% -63.8% 62.40 ± 50% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 477.83 ± 35% -47.1% 252.80 ± 18% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 723761 ± 35% -46.7% 385737 ± 23% perf-sched.wait_and_delay.count.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1896 ± 41% +204.0% 5763 ± 20% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 1712 ± 34% +181.2% 4814 ± 26% perf-sched.wait_and_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 3572 ± 20% +121.8% 7924 ± 15% perf-sched.wait_and_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 1736 ± 35% +158.6% 4491 ± 27% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 3117 ± 25% +161.0% 8134 ± 17% perf-sched.wait_and_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 12.30 ±215% -99.6% 0.05 ± 97% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 16.37 ± 22% +37.3% 22.47 ± 8% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.16 ±139% -97.3% 0.00 ±123% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 12.34 ± 34% +247.8% 42.92 ± 11% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 0.97 ±110% +3354.5% 33.55 ±162% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_blocks.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin 12.20 ± 94% -91.7% 1.02 ±131% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 31.17 ± 36% +399.8% 155.78 ± 15% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 1.54 ± 53% +393.0% 7.60 ± 35% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 2.13 ± 31% +53.9% 3.28 ± 12% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 14.25 ± 36% +718.5% 116.62 ± 13% perf-sched.wait_time.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 35.62 ± 54% +164.5% 94.19 ± 20% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 36.74 ± 44% +138.6% 87.65 ± 48% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 2.74 ± 32% +264.8% 10.00 ± 12% perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 510.34 ± 65% -92.2% 39.72 ± 47% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.shmem_alloc_folio 168.24 ±221% -99.9% 0.19 ±116% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 0.63 ±170% -99.3% 0.00 ±123% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 582.26 ± 22% +60.9% 936.72 ± 21% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 1044 ± 34% +199.8% 3131 ± 12% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 185.64 ±128% -98.9% 2.02 ±130% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1078 ± 23% +165.2% 2861 ± 28% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 196.06 ± 76% -95.1% 9.51 ±134% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown] 1918 ± 26% +114.1% 4107 ± 15% perf-sched.wait_time.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 941.82 ± 25% +144.5% 2303 ± 23% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 1971 ± 24% +119.6% 4329 ± 12% perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] *************************************************************************************************** lkp-spr-r02: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: gcc-12/performance/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-spr-r02/poll/stress-ng/60s commit: d2fc0ed52a ("Merge branch 'vfs-6.14.uncached_buffered_io'") aaec5a95d5 ("pipe_read: don't wake up the writer if the pipe is still full") d2fc0ed52a284a13 aaec5a95d59615523db03dd53c2 ---------------- --------------------------- %stddev %change %stddev \ | \ 4.049e+08 -54.4% 1.847e+08 cpuidle..usage 70.66 ± 2% -40.9% 41.74 ± 3% vmstat.procs.r 13673771 -55.5% 6089704 vmstat.system.cs 3388831 -9.5% 3067987 vmstat.system.in 6.62 +12.4 19.04 ± 6% mpstat.cpu.all.irq% 0.07 -0.0 0.06 mpstat.cpu.all.soft% 17.51 -6.8 10.76 mpstat.cpu.all.sys% 4.98 -1.0 4.01 ± 2% mpstat.cpu.all.usr% 18.00 ± 4% +110.2% 37.83 ± 37% mpstat.max_utilization.seconds 3483 ± 17% -38.5% 2141 ± 9% perf-c2c.DRAM.local 2266 ± 5% +256.6% 8081 ± 5% perf-c2c.DRAM.remote 173996 ± 2% -50.8% 85520 ± 12% perf-c2c.HITM.local 1097 ± 10% +97.5% 2168 ± 5% perf-c2c.HITM.remote 175093 ± 2% -49.9% 87689 ± 11% perf-c2c.HITM.total 2643650 +5.6% 2790795 ± 2% proc-vmstat.nr_active_anon 3308720 +4.6% 3459916 proc-vmstat.nr_file_pages 2427405 +6.2% 2578600 ± 2% proc-vmstat.nr_shmem 2643650 +5.6% 2790795 ± 2% proc-vmstat.nr_zone_active_anon 235308 -19.3% 189819 ± 12% proc-vmstat.numa_hint_faults_local 1437439 -5.5% 1358004 ± 3% proc-vmstat.pgfault 8.71e+08 -11.1% 7.745e+08 stress-ng.poll.ops 14516970 -11.1% 12907569 stress-ng.poll.ops_per_sec 181583 -57.1% 77818 ± 21% stress-ng.time.involuntary_context_switches 85474 +1.6% 86823 stress-ng.time.minor_page_faults 6150 -47.8% 3208 stress-ng.time.percent_of_cpu_this_job_got 2993 -50.6% 1477 stress-ng.time.system_time 711.20 -36.0% 454.85 stress-ng.time.user_time 4.427e+08 -56.2% 1.937e+08 stress-ng.time.voluntary_context_switches 834292 ± 4% -60.5% 329635 ± 12% sched_debug.cfs_rq:/.avg_vruntime.avg 520206 ± 5% -70.0% 155956 ± 44% sched_debug.cfs_rq:/.avg_vruntime.min 80954 ± 25% -78.0% 17846 ± 55% sched_debug.cfs_rq:/.left_deadline.avg 312463 ± 46% -69.5% 95397 ± 67% sched_debug.cfs_rq:/.left_deadline.stddev 80943 ± 25% -78.0% 17842 ± 55% sched_debug.cfs_rq:/.left_vruntime.avg 312436 ± 46% -69.5% 95382 ± 67% sched_debug.cfs_rq:/.left_vruntime.stddev 834292 ± 4% -60.5% 329635 ± 12% sched_debug.cfs_rq:/.min_vruntime.avg 520206 ± 5% -70.0% 155956 ± 44% sched_debug.cfs_rq:/.min_vruntime.min 80943 ± 25% -78.0% 17842 ± 55% sched_debug.cfs_rq:/.right_vruntime.avg 312436 ± 46% -69.5% 95382 ± 67% sched_debug.cfs_rq:/.right_vruntime.stddev 224.99 ± 4% -32.0% 152.91 ± 9% sched_debug.cfs_rq:/.runnable_avg.avg 212.19 ± 4% -30.4% 147.67 ± 8% sched_debug.cfs_rq:/.util_avg.avg 28.51 ± 5% -26.3% 21.02 ± 29% sched_debug.cfs_rq:/.util_est.avg 0.18 ± 6% -41.8% 0.11 ± 30% sched_debug.cpu.nr_running.avg 0.38 ± 6% -23.2% 0.29 ± 14% sched_debug.cpu.nr_running.stddev 1893149 -63.0% 700843 ± 44% sched_debug.cpu.nr_switches.avg 2007234 -62.4% 755101 ± 43% sched_debug.cpu.nr_switches.max 845934 ± 16% -65.9% 288868 ± 45% sched_debug.cpu.nr_switches.min 136058 ± 6% -62.3% 51280 ± 45% sched_debug.cpu.nr_switches.stddev 54.75 ± 29% -35.9% 35.08 ± 21% sched_debug.cpu.nr_uninterruptible.max 0.13 ± 3% +138.2% 0.30 ± 3% perf-stat.i.MPKI 4.279e+10 -24.6% 3.224e+10 perf-stat.i.branch-instructions 0.57 -0.1 0.47 ± 2% perf-stat.i.branch-miss-rate% 2.343e+08 -36.3% 1.493e+08 ± 2% perf-stat.i.branch-misses 3.22 ± 2% +7.6 10.77 ± 2% perf-stat.i.cache-miss-rate% 29560239 ± 2% +71.2% 50594050 ± 4% perf-stat.i.cache-misses 8.946e+08 -49.8% 4.494e+08 ± 2% perf-stat.i.cache-references 14167388 -55.5% 6301671 perf-stat.i.context-switches 1.33 +45.2% 1.93 ± 3% perf-stat.i.cpi 2.778e+11 +8.5% 3.013e+11 ± 2% perf-stat.i.cpu-cycles 2501362 -83.7% 408632 perf-stat.i.cpu-migrations 15733 ± 3% -52.7% 7444 ± 7% perf-stat.i.cycles-between-cache-misses 2.138e+11 -25.5% 1.594e+11 perf-stat.i.instructions 0.76 -29.3% 0.54 ± 2% perf-stat.i.ipc 74.48 -59.7% 29.98 perf-stat.i.metric.K/sec 21318 -6.8% 19869 ± 6% perf-stat.i.minor-faults 21318 -6.8% 19869 ± 6% perf-stat.i.page-faults 0.14 ± 2% +130.0% 0.32 ± 4% perf-stat.overall.MPKI 0.55 -0.1 0.46 ± 2% perf-stat.overall.branch-miss-rate% 3.30 +7.9 11.24 ± 2% perf-stat.overall.cache-miss-rate% 1.30 +45.5% 1.89 ± 3% perf-stat.overall.cpi 9426 ± 2% -36.7% 5969 ± 5% perf-stat.overall.cycles-between-cache-misses 0.77 -31.2% 0.53 ± 3% perf-stat.overall.ipc 4.206e+10 -24.7% 3.167e+10 perf-stat.ps.branch-instructions 2.299e+08 -36.3% 1.464e+08 ± 2% perf-stat.ps.branch-misses 28994632 ± 2% +71.3% 49675855 ± 4% perf-stat.ps.cache-misses 8.795e+08 -49.8% 4.419e+08 ± 2% perf-stat.ps.cache-references 13938322 -55.5% 6196929 perf-stat.ps.context-switches 2.732e+11 +8.3% 2.96e+11 ± 2% perf-stat.ps.cpu-cycles 2460505 -83.7% 402100 perf-stat.ps.cpu-migrations 2.102e+11 -25.5% 1.565e+11 perf-stat.ps.instructions 0.01 ± 82% +259.6% 0.05 ± 57% perf-stat.ps.major-faults 20785 -7.2% 19284 ± 6% perf-stat.ps.minor-faults 20785 -7.2% 19284 ± 6% perf-stat.ps.page-faults 1.283e+13 -25.5% 9.55e+12 perf-stat.total.instructions 40.62 -19.6 21.06 ± 10% perf-profile.calltrace.cycles-pp.stress_run 15.66 -8.0 7.64 ± 9% perf-profile.calltrace.cycles-pp.stress_poll.stress_run 12.84 -7.7 5.14 ± 16% perf-profile.calltrace.cycles-pp.write.stress_run 12.48 -7.5 4.98 ± 10% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 13.32 -7.3 6.04 ± 9% perf-profile.calltrace.cycles-pp.read.stress_poll.stress_run 11.55 -7.1 4.46 ± 17% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.stress_run 11.38 -7.0 4.33 ± 16% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.stress_run 12.17 -7.0 5.16 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read.stress_poll.stress_run 12.00 -7.0 5.02 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.stress_poll.stress_run 10.32 -6.7 3.60 ± 17% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.stress_run 10.86 -6.7 4.16 ± 9% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.stress_poll 10.38 -6.6 3.81 ± 9% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 9.91 -6.4 3.52 ± 8% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 7.32 -6.3 0.97 ± 3% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64 9.45 -6.3 3.13 ± 9% perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.21 -6.2 3.00 ± 8% perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.76 ± 2% -5.8 0.91 ± 3% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary 6.30 ± 2% -5.4 0.88 ± 3% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry 5.61 ± 2% -4.8 0.84 ± 3% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle 4.97 ± 2% -4.2 0.78 ± 3% perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue 4.84 ± 3% -4.1 0.77 ± 3% perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue 3.16 ± 4% -2.6 0.61 ± 5% perf-profile.calltrace.cycles-pp.dl_server_start.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending 2.57 -1.4 1.13 ± 7% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule 3.45 -1.2 2.20 ± 8% perf-profile.calltrace.cycles-pp.core_sys_select.do_pselect.__x64_sys_pselect6.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.70 -1.2 1.50 ± 8% perf-profile.calltrace.cycles-pp.__poll.stress_run 2.45 -1.1 1.31 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__poll.stress_run 2.43 -1.1 1.30 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll.stress_run 2.77 -1.1 1.65 ± 8% perf-profile.calltrace.cycles-pp.do_select.core_sys_select.do_pselect.__x64_sys_pselect6.do_syscall_64 3.70 -1.1 2.58 ± 8% perf-profile.calltrace.cycles-pp.ppoll.stress_run 2.92 -1.1 1.82 ± 8% perf-profile.calltrace.cycles-pp.__select 2.30 -1.1 1.20 ± 8% perf-profile.calltrace.cycles-pp.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll.stress_run 2.13 -1.1 1.08 ± 8% perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe.__poll 2.67 -1.0 1.63 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__select 2.65 -1.0 1.62 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select 2.54 -1.0 1.54 ± 8% perf-profile.calltrace.cycles-pp.__x64_sys_pselect6.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select 2.51 -1.0 1.51 ± 9% perf-profile.calltrace.cycles-pp.do_pselect.__x64_sys_pselect6.do_syscall_64.entry_SYSCALL_64_after_hwframe.__select 2.91 -0.9 2.02 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ppoll.stress_run 2.86 -0.9 1.98 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ppoll.stress_run 1.35 -0.9 0.49 ± 45% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 2.53 -0.8 1.72 ± 8% perf-profile.calltrace.cycles-pp.__x64_sys_ppoll.do_syscall_64.entry_SYSCALL_64_after_hwframe.ppoll.stress_run 1.07 -0.7 0.38 ± 70% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.vfs_read.ksys_read.do_syscall_64 2.01 -0.6 1.36 ± 28% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write.stress_run 2.57 -0.6 1.94 ± 9% perf-profile.calltrace.cycles-pp.pselect.stress_run 1.24 -0.6 0.64 ± 9% perf-profile.calltrace.cycles-pp.do_poll.do_sys_poll.__x64_sys_poll.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.32 -0.6 1.75 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.pselect.stress_run 2.30 -0.6 1.74 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.pselect.stress_run 2.18 -0.5 1.65 ± 9% perf-profile.calltrace.cycles-pp.__x64_sys_pselect6.do_syscall_64.entry_SYSCALL_64_after_hwframe.pselect.stress_run 2.10 -0.5 1.58 ± 9% perf-profile.calltrace.cycles-pp.do_pselect.__x64_sys_pselect6.do_syscall_64.entry_SYSCALL_64_after_hwframe.pselect 1.59 -0.5 1.08 ± 29% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read.stress_poll.stress_run 0.72 -0.5 0.26 ±100% perf-profile.calltrace.cycles-pp.__getrlimit.stress_run 1.46 -0.5 1.00 ± 9% perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.vfs_read.ksys_read.do_syscall_64 1.30 -0.4 0.86 ± 9% perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read.ksys_read 0.67 -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.__wake_up_sync_key.pipe_write.vfs_write.ksys_write.do_syscall_64 0.65 -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.stress_run 0.65 -0.4 0.26 ±100% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.stress_poll 0.77 -0.4 0.40 ± 71% perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write 0.93 -0.3 0.58 ± 45% perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64 0.70 -0.3 0.36 ± 70% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.setrlimit64.stress_run 0.62 ± 3% -0.3 0.29 ±100% perf-profile.calltrace.cycles-pp.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule.schedule 0.60 ± 4% -0.3 0.28 ±100% perf-profile.calltrace.cycles-pp.dequeue_entities.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule 1.09 -0.3 0.77 ± 8% perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.vfs_read.ksys_read.do_syscall_64 0.56 ± 4% -0.3 0.27 ±100% perf-profile.calltrace.cycles-pp.dl_server_stop.dequeue_entities.pick_task_fair.pick_next_task_fair.__pick_next_task 1.15 -0.3 0.87 ± 8% perf-profile.calltrace.cycles-pp.setrlimit64.stress_run 0.74 -0.3 0.47 ± 45% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.setrlimit64.stress_run 0.94 -0.3 0.68 ± 9% perf-profile.calltrace.cycles-pp.do_sys_poll.__x64_sys_ppoll.do_syscall_64.entry_SYSCALL_64_after_hwframe.ppoll 0.88 -0.2 0.65 ± 8% perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.pipe_read.vfs_read.ksys_read 0.95 -0.1 0.88 ± 5% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events 0.95 -0.1 0.88 ± 5% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output 0.96 -0.1 0.90 ± 5% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record 0.97 -0.1 0.91 ± 5% perf-profile.calltrace.cycles-pp.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record 0.97 -0.1 0.91 ± 5% perf-profile.calltrace.cycles-pp.__cmd_record 0.97 -0.1 0.91 ± 5% perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.__cmd_record 0.97 -0.1 0.91 ± 5% perf-profile.calltrace.cycles-pp.record__finish_output.__cmd_record 0.00 +0.7 0.65 ± 10% perf-profile.calltrace.cycles-pp.timerqueue_add.enqueue_hrtimer.__hrtimer_start_range_ns.hrtimer_start_range_ns.start_dl_timer 0.00 +0.7 0.67 ± 9% perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_start_range_ns.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity 0.00 +0.7 0.70 ± 11% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle 1.11 ± 8% +0.7 1.82 ± 5% perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.do_nanosleep.hrtimer_nanosleep 0.99 ± 9% +0.7 1.72 ± 6% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.do_nanosleep 0.00 +0.7 0.74 ± 11% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair 0.00 +0.8 0.75 ± 11% perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task 0.40 ± 71% +0.8 1.15 ± 9% perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule 0.00 +1.0 0.95 ± 10% perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule 0.08 ±223% +1.0 1.05 ± 10% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate.try_to_wake_up 0.75 ± 2% +3.1 3.88 ± 26% perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule_idle.do_idle.cpu_startup_entry 0.09 ±223% +3.7 3.83 ± 27% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule.schedule_idle.do_idle 0.00 +3.8 3.77 ± 27% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch 0.00 +3.8 3.78 ± 27% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule 0.00 +3.8 3.80 ± 27% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.finish_task_switch.__schedule.schedule_idle 39.66 +6.2 45.84 ± 2% perf-profile.calltrace.cycles-pp.common_startup_64 39.42 +6.2 45.61 ± 3% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64 39.39 +6.2 45.60 ± 3% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64 39.32 +6.3 45.57 ± 3% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64 3.96 ± 4% +8.8 12.76 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__hrtimer_start_range_ns.hrtimer_start_range_ns.start_dl_timer 4.16 ± 4% +8.9 13.02 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock.__hrtimer_start_range_ns.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity 5.00 ± 3% +9.2 14.23 perf-profile.calltrace.cycles-pp.__hrtimer_start_range_ns.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity.dl_server_start 3.14 ± 8% +12.2 15.34 ± 16% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity 3.33 ± 8% +12.4 15.73 ± 15% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity.dl_server_start 26.25 +12.6 38.87 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64 24.91 +13.0 37.86 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 24.29 +13.0 37.33 ± 3% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 8.54 +14.8 23.30 ± 4% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule 13.88 +15.3 29.13 ± 2% perf-profile.calltrace.cycles-pp.clock_nanosleep 12.88 +15.6 28.47 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 12.90 +15.6 28.49 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep 4.28 ± 10% +16.2 20.46 ± 7% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.hrtimer_try_to_cancel.dl_server_stop.dequeue_entities 5.70 ± 3% +16.3 22.00 ± 5% perf-profile.calltrace.cycles-pp.dl_server_stop.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule 4.20 ± 4% +16.4 20.62 ± 6% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.hrtimer_try_to_cancel.dl_server_stop.dequeue_entities.dequeue_task_fair 11.34 +16.5 27.80 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 11.04 +16.5 27.58 ± 3% perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 11.03 +16.5 27.57 ± 3% perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.88 +16.6 27.46 ± 3% perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64 4.56 ± 4% +16.7 21.23 ± 6% perf-profile.calltrace.cycles-pp.hrtimer_try_to_cancel.dl_server_stop.dequeue_entities.dequeue_task_fair.try_to_block_task 6.60 ± 2% +16.8 23.41 ± 4% perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.do_nanosleep.hrtimer_nanosleep 6.56 ± 2% +16.8 23.38 ± 4% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.do_nanosleep 9.56 +16.9 26.44 ± 4% perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 9.47 +16.9 26.38 ± 4% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep 10.92 +20.9 31.83 ± 4% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 10.03 ± 2% +21.0 31.00 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 9.54 ± 2% +21.0 30.58 ± 5% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 9.38 ± 2% +21.0 30.43 ± 5% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 9.54 ± 4% +21.4 30.97 ± 8% perf-profile.calltrace.cycles-pp.enqueue_dl_entity.dl_server_start.enqueue_task_fair.enqueue_task.ttwu_do_activate 9.13 ± 4% +21.6 30.74 ± 8% perf-profile.calltrace.cycles-pp.start_dl_timer.enqueue_dl_entity.dl_server_start.enqueue_task_fair.enqueue_task 8.84 ± 4% +21.7 30.56 ± 8% perf-profile.calltrace.cycles-pp.hrtimer_start_range_ns.start_dl_timer.enqueue_dl_entity.dl_server_start.enqueue_task_fair 6.41 ± 3% +24.0 30.41 ± 8% perf-profile.calltrace.cycles-pp.dl_server_start.enqueue_task_fair.enqueue_task.ttwu_do_activate.try_to_wake_up 8.60 ± 2% +24.8 33.35 ± 7% perf-profile.calltrace.cycles-pp.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt 8.58 ± 2% +24.8 33.33 ± 7% perf-profile.calltrace.cycles-pp.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 7.13 ± 3% +24.8 31.91 ± 7% perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup 7.15 ± 3% +24.8 31.94 ± 7% perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues 7.19 ± 3% +24.8 31.99 ± 7% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.hrtimer_wakeup.__hrtimer_run_queues.hrtimer_interrupt 8.91 ± 2% +24.8 33.76 ± 7% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 40.62 -19.6 21.06 ± 10% perf-profile.children.cycles-pp.stress_run 16.04 -8.1 7.94 ± 9% perf-profile.children.cycles-pp.stress_poll 13.84 -7.6 6.21 ± 8% perf-profile.children.cycles-pp.write 14.59 -7.6 7.03 ± 9% perf-profile.children.cycles-pp.read 12.53 -7.5 4.99 ± 10% perf-profile.children.cycles-pp.intel_idle 10.91 -6.7 4.20 ± 9% perf-profile.children.cycles-pp.ksys_read 10.44 -6.6 3.85 ± 9% perf-profile.children.cycles-pp.vfs_read 10.41 -6.5 3.90 ± 8% perf-profile.children.cycles-pp.ksys_write 10.00 -6.5 3.53 ± 9% perf-profile.children.cycles-pp.pipe_read 7.51 -6.4 1.07 ± 3% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 7.42 -6.4 0.98 ± 4% perf-profile.children.cycles-pp.flush_smp_call_function_queue 10.00 -6.4 3.59 ± 8% perf-profile.children.cycles-pp.vfs_write 9.34 -6.3 3.08 ± 8% perf-profile.children.cycles-pp.pipe_write 6.96 ± 2% -5.9 1.03 ± 3% perf-profile.children.cycles-pp.sched_ttwu_pending 5.18 -4.7 0.53 ± 8% perf-profile.children.cycles-pp.__wake_up_sync_key 4.50 -4.4 0.12 ± 10% perf-profile.children.cycles-pp.__wake_up_common 48.46 -2.3 46.12 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 2.58 -2.3 0.30 ± 6% perf-profile.children.cycles-pp.select_task_rq 47.90 -2.2 45.72 perf-profile.children.cycles-pp.do_syscall_64 2.35 -2.1 0.27 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair 2.03 -1.8 0.22 ± 5% perf-profile.children.cycles-pp.select_idle_sibling 4.74 -1.5 3.19 ± 8% perf-profile.children.cycles-pp.__x64_sys_pselect6 4.63 -1.5 3.10 ± 8% perf-profile.children.cycles-pp.do_pselect 2.68 -1.5 1.17 ± 7% perf-profile.children.cycles-pp.dequeue_entity 3.11 -1.3 1.80 ± 8% perf-profile.children.cycles-pp.do_sys_poll 3.35 -1.3 2.05 ± 8% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 2.34 -1.3 1.07 ± 9% perf-profile.children.cycles-pp.pipe_poll 3.48 -1.3 2.22 ± 9% perf-profile.children.cycles-pp.core_sys_select 2.82 -1.2 1.60 ± 8% perf-profile.children.cycles-pp.__poll 4.00 -1.2 2.82 ± 8% perf-profile.children.cycles-pp.ppoll 3.05 -1.1 1.92 ± 8% perf-profile.children.cycles-pp.__select 2.82 -1.1 1.69 ± 8% perf-profile.children.cycles-pp.do_select 2.32 -1.1 1.22 ± 8% perf-profile.children.cycles-pp.__x64_sys_poll 1.17 -1.0 0.15 ± 6% perf-profile.children.cycles-pp.available_idle_cpu 1.85 -1.0 0.84 ± 7% perf-profile.children.cycles-pp.update_load_avg 1.05 -1.0 0.08 ± 10% perf-profile.children.cycles-pp.ttwu_queue_wakelist 1.12 -0.8 0.27 ± 8% perf-profile.children.cycles-pp.switch_mm_irqs_off 3.48 -0.8 2.67 ± 8% perf-profile.children.cycles-pp.entry_SYSCALL_64 3.11 ± 3% -0.8 2.30 ± 4% perf-profile.children.cycles-pp.__pick_next_task 2.55 -0.8 1.74 ± 8% perf-profile.children.cycles-pp.__x64_sys_ppoll 2.07 -0.8 1.27 ± 8% perf-profile.children.cycles-pp.enqueue_entity 1.85 -0.8 1.07 ± 8% perf-profile.children.cycles-pp.do_poll 0.77 -0.7 0.03 ± 70% perf-profile.children.cycles-pp.__smp_call_single_queue 1.75 -0.7 1.03 ± 9% perf-profile.children.cycles-pp.mutex_lock 2.84 ± 3% -0.7 2.14 ± 4% perf-profile.children.cycles-pp.pick_next_task_fair 0.88 -0.7 0.19 ± 3% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 2.67 -0.7 2.02 ± 8% perf-profile.children.cycles-pp.pselect 0.92 -0.6 0.31 ± 7% perf-profile.children.cycles-pp.prepare_task_switch 0.89 -0.6 0.28 ± 9% perf-profile.children.cycles-pp.__switch_to 0.68 -0.6 0.07 ± 8% perf-profile.children.cycles-pp.set_task_cpu 0.77 -0.6 0.17 ± 4% perf-profile.children.cycles-pp.sysvec_call_function_single 0.65 -0.6 0.07 ± 9% perf-profile.children.cycles-pp.sched_mm_cid_migrate_to 0.72 -0.6 0.16 ± 3% perf-profile.children.cycles-pp.__sysvec_call_function_single 1.13 ± 3% -0.6 0.57 ± 9% perf-profile.children.cycles-pp.pick_task_fair 0.86 ± 3% -0.5 0.32 ± 8% perf-profile.children.cycles-pp.update_cfs_group 0.76 -0.5 0.22 ± 6% perf-profile.children.cycles-pp.__switch_to_asm 1.57 -0.5 1.05 ± 8% perf-profile.children.cycles-pp._copy_from_user 1.28 -0.5 0.77 ± 8% perf-profile.children.cycles-pp.read_tsc 0.80 -0.5 0.32 ± 9% perf-profile.children.cycles-pp.__pollwait 1.50 -0.5 1.03 ± 9% perf-profile.children.cycles-pp.copy_page_to_iter 1.00 -0.5 0.52 ± 6% perf-profile.children.cycles-pp.update_curr 0.98 -0.5 0.51 ± 10% perf-profile.children.cycles-pp.ktime_get 1.31 -0.4 0.87 ± 9% perf-profile.children.cycles-pp._copy_to_iter 0.76 -0.4 0.34 ± 8% perf-profile.children.cycles-pp.update_rq_clock 0.64 -0.4 0.23 ± 10% perf-profile.children.cycles-pp.native_sched_clock 0.56 -0.4 0.16 ± 7% perf-profile.children.cycles-pp.switch_fpu_return 0.76 -0.4 0.37 ± 9% perf-profile.children.cycles-pp.add_wait_queue 0.64 -0.4 0.25 ± 8% perf-profile.children.cycles-pp.sched_clock_cpu 1.56 -0.3 1.22 ± 8% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.56 -0.3 0.22 ± 10% perf-profile.children.cycles-pp.sched_clock 0.47 -0.3 0.13 ± 8% perf-profile.children.cycles-pp.do_perf_trace_sched_wakeup_template 0.83 -0.3 0.49 ± 8% perf-profile.children.cycles-pp.set_user_sigmask 1.34 -0.3 1.02 ± 8% perf-profile.children.cycles-pp.setrlimit64 1.12 -0.3 0.80 ± 8% perf-profile.children.cycles-pp.touch_atime 0.40 -0.3 0.10 ± 8% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 0.86 -0.3 0.56 ± 8% perf-profile.children.cycles-pp.__do_sys_prlimit64 0.59 -0.3 0.31 ± 4% perf-profile.children.cycles-pp.set_next_task_fair 0.67 -0.3 0.40 ± 8% perf-profile.children.cycles-pp.fdget 0.56 -0.3 0.29 ± 4% perf-profile.children.cycles-pp.set_next_entity 0.45 -0.3 0.18 ± 8% perf-profile.children.cycles-pp.__update_load_avg_se 0.82 -0.3 0.55 ± 8% perf-profile.children.cycles-pp.__getrlimit 0.94 -0.2 0.70 ± 8% perf-profile.children.cycles-pp.atime_needs_update 0.54 -0.2 0.29 ± 9% perf-profile.children.cycles-pp.hrtimer_active 0.91 -0.2 0.66 ± 8% perf-profile.children.cycles-pp.ktime_get_ts64 0.95 -0.2 0.72 ± 8% perf-profile.children.cycles-pp.copy_page_from_iter 0.93 -0.2 0.70 ± 9% perf-profile.children.cycles-pp.poll_select_finish 0.29 -0.2 0.07 ± 10% perf-profile.children.cycles-pp.perf_tp_event 0.34 -0.2 0.13 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_enter 0.47 -0.2 0.26 ± 9% perf-profile.children.cycles-pp.__rseq_handle_notify_resume 0.64 ± 2% -0.2 0.44 ± 11% perf-profile.children.cycles-pp.get_nohz_timer_target 0.78 -0.2 0.58 ± 8% perf-profile.children.cycles-pp._copy_from_iter 0.28 ± 2% -0.2 0.09 ± 7% perf-profile.children.cycles-pp.___perf_sw_event 0.36 -0.2 0.18 ± 9% perf-profile.children.cycles-pp.rseq_ip_fixup 0.39 ± 3% -0.2 0.20 ± 11% perf-profile.children.cycles-pp.task_contending 0.67 -0.2 0.49 ± 8% perf-profile.children.cycles-pp.get_timespec64 0.54 -0.2 0.37 ± 8% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.63 -0.2 0.46 ± 9% perf-profile.children.cycles-pp.mutex_unlock 0.26 ± 2% -0.2 0.10 ± 11% perf-profile.children.cycles-pp.__get_user_8 0.27 ± 2% -0.2 0.11 ± 10% perf-profile.children.cycles-pp.rseq_get_rseq_cs 0.31 ± 2% -0.2 0.15 ± 8% perf-profile.children.cycles-pp.__nanosleep 0.76 -0.2 0.60 ± 8% perf-profile.children.cycles-pp.current_time 0.58 ± 3% -0.2 0.43 ± 8% perf-profile.children.cycles-pp.idle_cpu 0.25 -0.2 0.10 ± 11% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.32 -0.1 0.17 ± 7% perf-profile.children.cycles-pp.do_prlimit 0.48 -0.1 0.34 ± 9% perf-profile.children.cycles-pp.fdget_pos 0.59 -0.1 0.44 ± 8% perf-profile.children.cycles-pp.file_update_time 0.64 -0.1 0.50 ± 8% perf-profile.children.cycles-pp.poll_freewait 0.54 -0.1 0.40 ± 8% perf-profile.children.cycles-pp.select_estimate_accuracy 0.24 -0.1 0.11 ± 6% perf-profile.children.cycles-pp.sleep 0.35 -0.1 0.22 ± 8% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.49 -0.1 0.36 ± 9% perf-profile.children.cycles-pp.rw_verify_area 0.22 ± 2% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.reweight_entity 0.87 ± 2% -0.1 0.75 ± 7% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested 0.48 -0.1 0.36 ± 9% perf-profile.children.cycles-pp.inode_needs_update_time 0.17 -0.1 0.06 ± 11% perf-profile.children.cycles-pp.local_clock_noinstr 0.32 -0.1 0.21 ± 6% perf-profile.children.cycles-pp.__irq_exit_rcu 0.67 -0.1 0.57 ± 9% perf-profile.children.cycles-pp.clockevents_program_event 0.25 -0.1 0.14 ± 8% perf-profile.children.cycles-pp.update_rq_clock_task 0.13 ± 3% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.__rdgsbase_inactive 0.18 ± 2% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.update_entity_lag 0.38 -0.1 0.29 ± 9% perf-profile.children.cycles-pp.remove_wait_queue 0.48 -0.1 0.39 ± 7% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.29 ± 3% -0.1 0.20 ± 7% perf-profile.children.cycles-pp.tick_nohz_handler 0.12 -0.1 0.03 ± 70% perf-profile.children.cycles-pp.ct_kernel_enter 0.40 -0.1 0.32 ± 8% perf-profile.children.cycles-pp.native_irq_return_iret 0.14 -0.1 0.06 ± 8% perf-profile.children.cycles-pp.ct_idle_exit 0.28 -0.1 0.20 ± 8% perf-profile.children.cycles-pp.__set_current_blocked 0.44 ± 2% -0.1 0.36 ± 8% perf-profile.children.cycles-pp.x64_sys_call 0.12 ± 3% -0.1 0.04 ± 45% perf-profile.children.cycles-pp.__calc_delta 0.26 ± 2% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.update_process_times 0.32 -0.1 0.24 ± 8% perf-profile.children.cycles-pp._copy_to_user 0.32 -0.1 0.25 ± 9% perf-profile.children.cycles-pp.__cond_resched 0.32 -0.1 0.25 ± 8% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg 0.13 ± 3% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.__dequeue_entity 0.12 ± 3% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.call_cpuidle 0.28 -0.1 0.22 ± 9% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.28 -0.1 0.21 ± 7% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.18 ± 2% -0.1 0.12 ± 4% perf-profile.children.cycles-pp.handle_softirqs 0.95 -0.1 0.88 ± 5% perf-profile.children.cycles-pp.queue_event 0.12 -0.1 0.06 ± 8% perf-profile.children.cycles-pp.update_min_vruntime 0.95 -0.1 0.88 ± 5% perf-profile.children.cycles-pp.ordered_events__queue 0.30 -0.1 0.23 ± 9% perf-profile.children.cycles-pp.put_timespec64 0.20 ± 2% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.place_entity 0.96 -0.1 0.90 ± 5% perf-profile.children.cycles-pp.process_simple 0.10 ± 5% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.tick_nohz_stop_idle 0.97 -0.1 0.91 ± 5% perf-profile.children.cycles-pp.perf_session__process_events 0.97 -0.1 0.91 ± 5% perf-profile.children.cycles-pp.reader__read_event 0.97 -0.1 0.91 ± 5% perf-profile.children.cycles-pp.record__finish_output 0.47 -0.1 0.42 ± 9% perf-profile.children.cycles-pp.lapic_next_deadline 0.19 ± 2% -0.0 0.14 ± 10% perf-profile.children.cycles-pp.recalc_sigpending 0.20 -0.0 0.15 ± 9% perf-profile.children.cycles-pp.native_apic_msr_eoi 0.18 ± 4% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.__enqueue_entity 0.14 ± 2% -0.0 0.10 ± 8% perf-profile.children.cycles-pp.poll_select_set_timeout 0.10 -0.0 0.06 ± 13% perf-profile.children.cycles-pp.task_non_contending 0.15 -0.0 0.11 ± 10% perf-profile.children.cycles-pp.os_xsave 0.10 ± 3% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__mutex_lock 0.16 ± 2% -0.0 0.12 ± 9% perf-profile.children.cycles-pp.fput 0.19 -0.0 0.15 ± 9% perf-profile.children.cycles-pp.security_file_permission 0.15 -0.0 0.12 ± 8% perf-profile.children.cycles-pp.irqtime_account_irq 0.17 ± 2% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.rcu_all_qs 0.12 ± 3% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.vruntime_eligible 0.14 -0.0 0.11 ± 12% perf-profile.children.cycles-pp.__check_object_size 0.06 ± 7% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.__put_user_8 0.11 -0.0 0.08 ± 10% perf-profile.children.cycles-pp.__fdelt_warn 0.14 ± 3% -0.0 0.11 ± 9% perf-profile.children.cycles-pp.__memset 0.12 ± 4% -0.0 0.09 ± 10% perf-profile.children.cycles-pp.kill_fasync 0.14 -0.0 0.11 ± 11% perf-profile.children.cycles-pp.avg_vruntime 0.09 -0.0 0.06 ± 11% perf-profile.children.cycles-pp.task_mm_cid_work 0.06 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.make_vfsgid 0.09 ± 4% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.task_work_run 0.08 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__fdelt_chk@plt 0.10 ± 5% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.rseq_update_cpu_node_id 0.07 -0.0 0.05 perf-profile.children.cycles-pp.put_prev_entity 0.07 -0.0 0.05 ± 7% perf-profile.children.cycles-pp.update_irq_load_avg 0.07 -0.0 0.06 ± 9% perf-profile.children.cycles-pp.get_sigset_argpack 0.08 ± 4% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.put_prev_task_fair 0.07 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.sched_balance_domains 0.07 ± 9% +0.0 0.10 ± 6% perf-profile.children.cycles-pp._find_next_and_bit 0.06 ± 6% +0.0 0.09 ± 30% perf-profile.children.cycles-pp.handle_internal_command 0.06 ± 6% +0.0 0.09 ± 30% perf-profile.children.cycles-pp.main 0.06 ± 6% +0.0 0.09 ± 30% perf-profile.children.cycles-pp.run_builtin 0.05 +0.0 0.08 ± 31% perf-profile.children.cycles-pp.cmd_record 0.00 +0.1 0.06 ± 31% perf-profile.children.cycles-pp.perf_mmap__push 0.00 +0.1 0.07 ± 36% perf-profile.children.cycles-pp.record__mmap_read_evlist 0.48 +0.1 0.57 ± 8% perf-profile.children.cycles-pp.timerqueue_del 0.12 ± 3% +0.1 0.22 ± 19% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.00 +0.1 0.11 ± 8% perf-profile.children.cycles-pp.sched_balance_find_src_rq 0.41 +0.1 0.52 ± 8% perf-profile.children.cycles-pp.rb_erase 0.08 ± 6% +0.1 0.21 ± 6% perf-profile.children.cycles-pp.update_curr_dl_se 0.24 ± 4% +0.2 0.43 ± 19% perf-profile.children.cycles-pp.__get_next_timer_interrupt 0.30 ± 3% +0.2 0.48 ± 18% perf-profile.children.cycles-pp.tick_nohz_next_event 0.12 ± 7% +0.2 0.32 ± 22% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.45 ± 2% +0.3 0.74 ± 18% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.15 +0.3 0.46 ± 22% perf-profile.children.cycles-pp.poll_idle 0.80 ± 16% +0.4 1.17 ± 9% perf-profile.children.cycles-pp.sched_balance_newidle 0.38 ± 28% +0.4 0.77 ± 11% perf-profile.children.cycles-pp.update_sd_lb_stats 0.36 ± 30% +0.4 0.74 ± 11% perf-profile.children.cycles-pp.update_sg_lb_stats 0.40 ± 27% +0.4 0.79 ± 11% perf-profile.children.cycles-pp.sched_balance_find_src_group 0.47 ± 24% +0.5 1.00 ± 9% perf-profile.children.cycles-pp.sched_balance_rq 1.51 +2.6 4.10 ± 25% perf-profile.children.cycles-pp.finish_task_switch 39.66 +6.2 45.84 ± 2% perf-profile.children.cycles-pp.common_startup_64 39.66 +6.2 45.84 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 39.42 +6.2 45.61 ± 3% perf-profile.children.cycles-pp.start_secondary 39.60 +6.2 45.82 ± 2% perf-profile.children.cycles-pp.do_idle 5.60 ± 2% +8.7 14.34 perf-profile.children.cycles-pp._raw_spin_lock 6.49 ± 2% +8.8 15.24 perf-profile.children.cycles-pp.__hrtimer_start_range_ns 13.94 +12.5 26.47 ± 4% perf-profile.children.cycles-pp.schedule 26.42 +12.7 39.07 ± 3% perf-profile.children.cycles-pp.cpuidle_idle_call 25.01 +13.0 37.98 ± 3% perf-profile.children.cycles-pp.cpuidle_enter 24.98 +13.0 37.97 ± 3% perf-profile.children.cycles-pp.cpuidle_enter_state 18.29 +13.4 31.65 ± 6% perf-profile.children.cycles-pp.__schedule 9.51 ± 2% +14.3 23.82 ± 5% perf-profile.children.cycles-pp.dequeue_entities 8.69 +14.7 23.42 ± 4% perf-profile.children.cycles-pp.try_to_block_task 8.63 +14.8 23.40 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair 14.16 +15.1 29.27 ± 2% perf-profile.children.cycles-pp.clock_nanosleep 6.58 ± 3% +15.9 22.49 ± 5% perf-profile.children.cycles-pp.dl_server_stop 6.11 ± 3% +16.0 22.10 ± 6% perf-profile.children.cycles-pp.hrtimer_try_to_cancel 11.36 +16.4 27.80 ± 3% perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep 11.13 +16.5 27.63 ± 3% perf-profile.children.cycles-pp.common_nsleep 11.04 +16.5 27.58 ± 3% perf-profile.children.cycles-pp.hrtimer_nanosleep 10.90 +16.6 27.48 ± 3% perf-profile.children.cycles-pp.do_nanosleep 13.85 ± 2% +19.6 33.48 ± 7% perf-profile.children.cycles-pp.ttwu_do_activate 13.09 ± 2% +20.3 33.39 ± 7% perf-profile.children.cycles-pp.enqueue_task 13.56 +20.4 33.96 ± 7% perf-profile.children.cycles-pp.try_to_wake_up 12.92 ± 3% +20.4 33.35 ± 7% perf-profile.children.cycles-pp.enqueue_task_fair 10.20 ± 4% +21.4 31.56 ± 8% perf-profile.children.cycles-pp.enqueue_dl_entity 10.24 ± 4% +21.4 31.61 ± 8% perf-profile.children.cycles-pp.dl_server_start 11.02 ± 3% +21.5 32.51 ± 7% perf-profile.children.cycles-pp.hrtimer_start_range_ns 9.74 ± 4% +21.6 31.34 ± 8% perf-profile.children.cycles-pp.start_dl_timer 12.42 +24.0 36.41 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 11.69 +24.1 35.78 ± 6% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 11.11 ± 2% +24.2 35.30 ± 6% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 10.91 ± 2% +24.2 35.13 ± 6% perf-profile.children.cycles-pp.hrtimer_interrupt 10.24 ± 2% +24.2 34.49 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 9.69 ± 2% +24.3 33.96 ± 7% perf-profile.children.cycles-pp.hrtimer_wakeup 11.90 ± 4% +27.9 39.77 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 13.14 ± 4% +38.1 51.29 ± 8% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 12.52 -7.5 4.99 ± 10% perf-profile.self.cycles-pp.intel_idle 3.10 -1.4 1.72 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 1.16 -1.0 0.15 ± 6% perf-profile.self.cycles-pp.available_idle_cpu 1.65 -0.9 0.74 ± 5% perf-profile.self.cycles-pp.__schedule 1.07 -0.9 0.20 ± 8% perf-profile.self.cycles-pp.switch_mm_irqs_off 1.44 -0.6 0.80 ± 9% perf-profile.self.cycles-pp.mutex_lock 0.87 -0.6 0.27 ± 9% perf-profile.self.cycles-pp.__switch_to 0.88 -0.6 0.28 ± 8% perf-profile.self.cycles-pp.update_load_avg 1.94 -0.6 1.36 ± 8% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.64 -0.6 0.07 ± 9% perf-profile.self.cycles-pp.sched_mm_cid_migrate_to 0.85 ± 3% -0.5 0.31 ± 8% perf-profile.self.cycles-pp.update_cfs_group 0.75 -0.5 0.22 ± 6% perf-profile.self.cycles-pp.__switch_to_asm 1.51 -0.5 1.02 ± 8% perf-profile.self.cycles-pp._copy_from_user 0.72 -0.5 0.23 ± 6% perf-profile.self.cycles-pp.prepare_task_switch 0.58 -0.5 0.09 ± 7% perf-profile.self.cycles-pp.__wake_up_common 1.22 -0.5 0.74 ± 8% perf-profile.self.cycles-pp.read_tsc 0.77 -0.5 0.31 ± 9% perf-profile.self.cycles-pp.__pollwait 1.25 -0.5 0.79 ± 9% perf-profile.self.cycles-pp.stress_poll 1.29 ± 2% -0.5 0.83 ± 8% perf-profile.self.cycles-pp.pipe_read 1.26 -0.4 0.84 ± 9% perf-profile.self.cycles-pp._copy_to_iter 0.59 -0.4 0.18 ± 4% perf-profile.self.cycles-pp.finish_task_switch 0.62 -0.4 0.22 ± 9% perf-profile.self.cycles-pp.native_sched_clock 0.71 -0.4 0.34 ± 9% perf-profile.self.cycles-pp.pipe_poll 0.43 -0.4 0.08 ± 6% perf-profile.self.cycles-pp.try_to_wake_up 0.92 -0.4 0.57 ± 9% perf-profile.self.cycles-pp.pipe_write 1.53 -0.3 1.19 ± 8% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.40 -0.3 0.10 ± 8% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 0.42 -0.3 0.14 ± 8% perf-profile.self.cycles-pp.menu_select 0.56 -0.3 0.29 ± 9% perf-profile.self.cycles-pp.do_sys_poll 0.63 -0.3 0.37 ± 9% perf-profile.self.cycles-pp.fdget 1.08 -0.2 0.83 ± 9% perf-profile.self.cycles-pp.read 0.38 -0.2 0.13 ± 7% perf-profile.self.cycles-pp.dequeue_entity 0.42 -0.2 0.17 ± 8% perf-profile.self.cycles-pp.__update_load_avg_se 0.92 -0.2 0.67 ± 8% perf-profile.self.cycles-pp.do_syscall_64 0.99 -0.2 0.74 ± 8% perf-profile.self.cycles-pp.write 0.28 -0.2 0.04 ± 45% perf-profile.self.cycles-pp.set_task_cpu 0.51 -0.2 0.28 ± 10% perf-profile.self.cycles-pp.hrtimer_active 0.46 -0.2 0.23 ± 7% perf-profile.self.cycles-pp.update_rq_clock 0.76 -0.2 0.57 ± 9% perf-profile.self.cycles-pp._copy_from_iter 0.32 ± 2% -0.2 0.13 ± 8% perf-profile.self.cycles-pp.update_curr 0.70 -0.2 0.52 ± 8% perf-profile.self.cycles-pp.entry_SYSCALL_64 0.28 -0.2 0.11 ± 8% perf-profile.self.cycles-pp.do_idle 0.34 -0.2 0.17 ± 7% perf-profile.self.cycles-pp.clock_nanosleep 0.60 -0.2 0.44 ± 9% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.91 -0.2 0.74 ± 9% perf-profile.self.cycles-pp.cpuidle_enter_state 0.22 ± 3% -0.2 0.06 ± 7% perf-profile.self.cycles-pp.pick_next_task_fair 0.60 -0.2 0.44 ± 9% perf-profile.self.cycles-pp.mutex_unlock 0.57 ± 2% -0.2 0.41 ± 9% perf-profile.self.cycles-pp.idle_cpu 0.25 ± 2% -0.2 0.10 ± 9% perf-profile.self.cycles-pp.__get_user_8 0.45 ± 2% -0.2 0.30 ± 9% perf-profile.self.cycles-pp.atime_needs_update 0.37 ± 2% -0.1 0.23 ± 11% perf-profile.self.cycles-pp.ktime_get 0.58 -0.1 0.43 ± 9% perf-profile.self.cycles-pp.vfs_read 0.24 -0.1 0.09 ± 7% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.44 ± 2% -0.1 0.30 ± 9% perf-profile.self.cycles-pp.fdget_pos 0.46 -0.1 0.34 ± 9% perf-profile.self.cycles-pp.ppoll 0.21 -0.1 0.08 ± 10% perf-profile.self.cycles-pp.cpuidle_idle_call 0.23 ± 2% -0.1 0.10 ± 10% perf-profile.self.cycles-pp.sched_balance_newidle 0.19 ± 2% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.___perf_sw_event 0.18 ± 2% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.select_task_rq_fair 0.46 -0.1 0.33 ± 8% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.21 -0.1 0.09 ± 12% perf-profile.self.cycles-pp.sleep 0.57 -0.1 0.46 ± 9% perf-profile.self.cycles-pp.do_select 0.46 -0.1 0.34 ± 8% perf-profile.self.cycles-pp.vfs_write 0.42 -0.1 0.31 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.16 ± 2% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.switch_fpu_return 0.29 -0.1 0.18 ± 9% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.19 -0.1 0.08 ± 8% perf-profile.self.cycles-pp.reweight_entity 0.20 -0.1 0.10 ± 13% perf-profile.self.cycles-pp.hrtimer_start_range_ns 0.13 ± 3% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.__rdgsbase_inactive 0.29 -0.1 0.20 ± 10% perf-profile.self.cycles-pp.rw_verify_area 0.11 -0.1 0.02 ± 99% perf-profile.self.cycles-pp.__calc_delta 0.44 -0.1 0.35 ± 8% perf-profile.self.cycles-pp.current_time 0.40 -0.1 0.32 ± 8% perf-profile.self.cycles-pp.native_irq_return_iret 0.18 ± 2% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.update_rq_clock_task 0.11 ± 4% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.ttwu_do_activate 0.28 -0.1 0.20 ± 8% perf-profile.self.cycles-pp.__do_sys_prlimit64 0.10 ± 3% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.__dequeue_entity 0.30 -0.1 0.23 ± 5% perf-profile.self.cycles-pp.enqueue_task_fair 0.31 -0.1 0.24 ± 9% perf-profile.self.cycles-pp._copy_to_user 0.39 -0.1 0.32 ± 9% perf-profile.self.cycles-pp.x64_sys_call 0.11 -0.1 0.04 ± 44% perf-profile.self.cycles-pp.update_min_vruntime 0.28 -0.1 0.21 ± 9% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.29 ± 2% -0.1 0.22 ± 8% perf-profile.self.cycles-pp.ktime_get_ts64 0.28 -0.1 0.22 ± 9% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg 0.23 -0.1 0.17 ± 7% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.10 ± 3% -0.1 0.04 ± 45% perf-profile.self.cycles-pp.place_entity 0.13 ± 3% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.__wake_up_sync_key 0.20 ± 2% -0.1 0.14 ± 9% perf-profile.self.cycles-pp.select_estimate_accuracy 0.47 -0.1 0.42 ± 9% perf-profile.self.cycles-pp.lapic_next_deadline 0.30 -0.1 0.25 ± 11% perf-profile.self.cycles-pp.core_sys_select 0.21 ± 3% -0.1 0.16 ± 8% perf-profile.self.cycles-pp.dequeue_entities 0.20 -0.1 0.15 ± 7% perf-profile.self.cycles-pp.native_apic_msr_eoi 0.15 -0.1 0.10 ± 10% perf-profile.self.cycles-pp.get_timespec64 0.24 -0.0 0.19 ± 7% perf-profile.self.cycles-pp.do_poll 0.21 -0.0 0.16 ± 9% perf-profile.self.cycles-pp.setrlimit64 0.21 -0.0 0.16 ± 7% perf-profile.self.cycles-pp.ksys_write 0.19 ± 2% -0.0 0.14 ± 10% perf-profile.self.cycles-pp.recalc_sigpending 0.23 ± 2% -0.0 0.18 ± 8% perf-profile.self.cycles-pp.ksys_read 0.18 -0.0 0.14 ± 10% perf-profile.self.cycles-pp.__cond_resched 0.17 ± 2% -0.0 0.13 ± 8% perf-profile.self.cycles-pp.copy_page_from_iter 0.15 -0.0 0.11 ± 9% perf-profile.self.cycles-pp.os_xsave 0.11 ± 6% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.__pick_next_task 0.16 ± 2% -0.0 0.12 ± 9% perf-profile.self.cycles-pp.__select 0.21 -0.0 0.17 ± 9% perf-profile.self.cycles-pp.copy_page_to_iter 0.17 ± 4% -0.0 0.13 ± 8% perf-profile.self.cycles-pp.__enqueue_entity 0.06 -0.0 0.02 ± 99% perf-profile.self.cycles-pp.__put_user_8 0.06 -0.0 0.02 ± 99% perf-profile.self.cycles-pp.update_irq_load_avg 0.09 ± 5% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.common_nsleep 0.16 ± 3% -0.0 0.12 ± 7% perf-profile.self.cycles-pp.nohz_run_idle_balance 0.12 ± 3% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.inode_needs_update_time 0.16 -0.0 0.13 ± 8% perf-profile.self.cycles-pp.security_file_permission 0.15 ± 2% -0.0 0.12 ± 9% perf-profile.self.cycles-pp.fput 0.11 -0.0 0.08 ± 7% perf-profile.self.cycles-pp.vruntime_eligible 0.12 ± 3% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.__nanosleep 0.12 -0.0 0.09 ± 9% perf-profile.self.cycles-pp.__x64_sys_clock_nanosleep 0.07 -0.0 0.04 ± 44% perf-profile.self.cycles-pp.__fdelt_warn 0.13 ± 2% -0.0 0.10 ± 11% perf-profile.self.cycles-pp.__poll 0.12 -0.0 0.09 ± 9% perf-profile.self.cycles-pp.poll_select_finish 0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.set_user_sigmask 0.13 ± 2% -0.0 0.10 ± 9% perf-profile.self.cycles-pp.pselect 0.10 ± 4% -0.0 0.08 ± 13% perf-profile.self.cycles-pp.get_nohz_timer_target 0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.__getrlimit 0.09 -0.0 0.07 ± 7% perf-profile.self.cycles-pp.do_prlimit 0.11 ± 4% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.rcu_all_qs 0.09 -0.0 0.07 ± 7% perf-profile.self.cycles-pp.rseq_update_cpu_node_id 0.07 ± 5% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.get_sigset_argpack 0.10 ± 3% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.do_nanosleep 0.10 ± 4% -0.0 0.08 ± 7% perf-profile.self.cycles-pp.file_update_time 0.07 ± 7% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.kill_fasync 0.12 ± 3% -0.0 0.10 ± 9% perf-profile.self.cycles-pp.__memset 0.10 ± 5% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.start_dl_timer 0.08 ± 4% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.task_mm_cid_work 0.06 -0.0 0.04 ± 44% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.09 -0.0 0.07 ± 10% perf-profile.self.cycles-pp.__x64_sys_ppoll 0.13 ± 3% -0.0 0.11 ± 7% perf-profile.self.cycles-pp.avg_vruntime 0.08 -0.0 0.06 ± 11% perf-profile.self.cycles-pp.add_wait_queue 0.08 -0.0 0.06 ± 11% perf-profile.self.cycles-pp.hrtimer_nanosleep 0.08 -0.0 0.06 ± 7% perf-profile.self.cycles-pp.do_pselect 0.07 -0.0 0.06 ± 8% perf-profile.self.cycles-pp.touch_atime 0.07 ± 7% +0.0 0.09 ± 6% perf-profile.self.cycles-pp._find_next_and_bit 0.00 +0.1 0.11 ± 6% perf-profile.self.cycles-pp.sched_balance_find_src_rq 0.40 +0.1 0.52 ± 8% perf-profile.self.cycles-pp.rb_erase 0.00 +0.2 0.15 ± 9% perf-profile.self.cycles-pp.update_curr_dl_se 0.29 ± 30% +0.3 0.58 ± 11% perf-profile.self.cycles-pp.update_sg_lb_stats 13.13 ± 5% +38.1 51.28 ± 8% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-icl-2sp8: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: gcc-12/performance/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/tee/stress-ng/60s commit: d2fc0ed52a ("Merge branch 'vfs-6.14.uncached_buffered_io'") aaec5a95d5 ("pipe_read: don't wake up the writer if the pipe is still full") d2fc0ed52a284a13 aaec5a95d59615523db03dd53c2 ---------------- --------------------------- %stddev %change %stddev \ | \ 123346 ± 30% -48.2% 63936 ± 24% cpuidle..usage 11.44 +5.9 17.37 mpstat.cpu.all.usr% 1237 ± 32% -83.3% 206.40 ± 20% perf-c2c.DRAM.local 934.33 ± 7% +53.9% 1438 ± 10% perf-c2c.DRAM.remote 263.67 ± 11% +26.7% 334.00 ± 5% perf-c2c.HITM.remote 11.14 +51.4% 16.87 vmstat.cpu.us 22963300 -95.8% 968409 vmstat.system.cs 197683 -8.3% 181346 vmstat.system.in 5623585 ± 19% -55.7% 2490490 ± 19% numa-meminfo.node1.Active 5623585 ± 19% -55.7% 2490490 ± 19% numa-meminfo.node1.Active(anon) 1738706 ± 8% -70.3% 516074 ± 29% numa-meminfo.node1.Mapped 5336025 ± 20% -61.3% 2063073 ± 14% numa-meminfo.node1.Shmem 7701721 +79.2% 13801344 numa-numastat.node0.local_node 7732133 +79.0% 13842144 numa-numastat.node0.numa_hit 9660861 ± 4% +51.6% 14643708 numa-numastat.node1.local_node 9696676 ± 4% +51.3% 14669530 numa-numastat.node1.numa_hit 4203288 +500.7% 25250189 stress-ng.tee.ops 70053 +500.7% 420829 stress-ng.tee.ops_per_sec 7.264e+08 -96.5% 25607922 stress-ng.time.involuntary_context_switches 3392 -7.6% 3134 stress-ng.time.system_time 356.72 +78.5% 636.76 stress-ng.time.user_time 7.31e+08 -95.1% 35783971 stress-ng.time.voluntary_context_switches 7716552 +79.4% 13841559 numa-vmstat.node0.numa_hit 7686141 +79.6% 13800760 numa-vmstat.node0.numa_local 1403134 ± 19% -55.6% 622772 ± 19% numa-vmstat.node1.nr_active_anon 432841 ± 7% -69.9% 130433 ± 27% numa-vmstat.node1.nr_mapped 1331229 ± 19% -61.3% 515736 ± 14% numa-vmstat.node1.nr_shmem 1403134 ± 19% -55.6% 622771 ± 19% numa-vmstat.node1.nr_zone_active_anon 9682370 ± 3% +51.5% 14668660 numa-vmstat.node1.numa_hit 9646555 ± 4% +51.8% 14642837 numa-vmstat.node1.numa_local 6151580 ± 17% -51.1% 3005085 ± 4% meminfo.Active 6151580 ± 17% -51.1% 3005085 ± 4% meminfo.Active(anon) 8920705 ± 11% -35.0% 5794702 meminfo.Cached 7882102 ± 13% -40.6% 4678787 meminfo.Committed_AS 1893975 ± 6% -63.9% 682914 ± 24% meminfo.Mapped 11167190 ± 9% -28.6% 7968104 meminfo.Memused 27261 ± 2% -13.4% 23618 ± 6% meminfo.PageTables 5395896 ± 19% -57.9% 2269891 meminfo.Shmem 11232918 ± 9% -27.7% 8115848 meminfo.max_used_kB 0.55 ± 5% -24.7% 0.42 ± 4% sched_debug.cfs_rq:/.h_nr_running.stddev 267.15 ± 77% -71.5% 76.26 ± 16% sched_debug.cfs_rq:/.load_avg.avg 10410 ±122% -93.8% 641.30 ± 33% sched_debug.cfs_rq:/.load_avg.max 1371 ±112% -88.1% 163.88 ± 10% sched_debug.cfs_rq:/.load_avg.stddev 833.50 ± 23% +24.0% 1033 sched_debug.cfs_rq:/.runnable_avg.min 247.56 ± 10% -14.6% 211.50 ± 12% sched_debug.cfs_rq:/.runnable_avg.stddev 485.86 ± 5% +30.5% 633.96 ± 5% sched_debug.cfs_rq:/.util_est.avg 1085 ± 5% +16.3% 1262 ± 9% sched_debug.cfs_rq:/.util_est.max 173.70 ± 7% +36.8% 237.65 ± 6% sched_debug.cfs_rq:/.util_est.stddev 398375 ± 4% +12.5% 448175 ± 2% sched_debug.cpu.avg_idle.avg 176609 ± 11% +38.4% 244462 ± 5% sched_debug.cpu.avg_idle.stddev 0.55 ± 5% -22.3% 0.43 ± 3% sched_debug.cpu.nr_running.stddev 11181287 -95.8% 474442 sched_debug.cpu.nr_switches.avg 11740535 -95.7% 505782 ± 2% sched_debug.cpu.nr_switches.max 8593260 ± 5% -95.1% 424452 ± 4% sched_debug.cpu.nr_switches.min 569340 ± 10% -97.7% 12920 ± 14% sched_debug.cpu.nr_switches.stddev 1537988 ± 17% -51.2% 751031 ± 3% proc-vmstat.nr_active_anon 6276579 +1.3% 6356360 proc-vmstat.nr_dirty_background_threshold 12568504 +1.3% 12728261 proc-vmstat.nr_dirty_threshold 2230307 ± 11% -35.1% 1448223 proc-vmstat.nr_file_pages 63142546 +1.3% 63941500 proc-vmstat.nr_free_pages 472553 ± 6% -64.0% 170048 ± 22% proc-vmstat.nr_mapped 6828 ± 2% -13.2% 5929 ± 6% proc-vmstat.nr_page_table_pages 1349103 ± 19% -58.0% 567019 proc-vmstat.nr_shmem 26209 ± 2% -6.9% 24399 proc-vmstat.nr_slab_reclaimable 1537988 ± 17% -51.2% 751031 ± 3% proc-vmstat.nr_zone_active_anon 167438 ± 12% -50.9% 82288 ± 24% proc-vmstat.numa_hint_faults 106871 ± 19% -62.1% 40516 ± 33% proc-vmstat.numa_hint_faults_local 17429790 ± 2% +63.6% 28512466 proc-vmstat.numa_hit 17363565 ± 2% +63.8% 28445844 proc-vmstat.numa_local 17470213 ± 2% +63.5% 28565863 proc-vmstat.pgalloc_normal 645222 -20.6% 512041 ± 3% proc-vmstat.pgfault 15401346 +80.1% 27730965 proc-vmstat.pgfree 5.248e+10 +8.6% 5.701e+10 perf-stat.i.branch-instructions 0.33 ± 3% -0.2 0.10 ± 11% perf-stat.i.branch-miss-rate% 1.647e+08 ± 2% -71.0% 47792335 ± 9% perf-stat.i.branch-misses 19.40 ± 8% +5.2 24.63 ± 7% perf-stat.i.cache-miss-rate% 9432268 ± 15% -23.5% 7215380 ± 14% perf-stat.i.cache-misses 46766538 ± 10% -34.7% 30557228 ± 8% perf-stat.i.cache-references 23928853 -95.8% 1009521 perf-stat.i.context-switches 373.88 -95.8% 15.81 perf-stat.i.metric.K/sec 8711 ± 3% -23.2% 6694 ± 4% perf-stat.i.minor-faults 8711 ± 3% -23.1% 6694 ± 4% perf-stat.i.page-faults 0.31 ± 2% -0.2 0.08 ± 9% perf-stat.overall.branch-miss-rate% 20.07 ± 7% +3.3 23.33 ± 6% perf-stat.overall.cache-miss-rate% 5.16e+10 +8.6% 5.606e+10 perf-stat.ps.branch-instructions 1.616e+08 ± 2% -71.0% 46862413 ± 9% perf-stat.ps.branch-misses 9324763 ± 15% -23.8% 7101263 ± 15% perf-stat.ps.cache-misses 46321216 ± 10% -34.6% 30283901 ± 8% perf-stat.ps.cache-references 23514896 -95.8% 992706 perf-stat.ps.context-switches 8550 ± 2% -23.0% 6584 ± 4% perf-stat.ps.minor-faults 8550 ± 2% -23.0% 6584 ± 4% perf-stat.ps.page-faults 0.00 ±223% +2945.0% 0.08 ±150% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 13.16 ± 77% -93.4% 0.87 ±132% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.00 ±223% +74228.0% 0.62 ± 53% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 0.98 ± 30% -70.0% 0.30 ± 17% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.18 ±140% +313.8% 0.76 ± 45% perf-sched.sch_delay.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 14.41 ±148% -99.1% 0.12 ± 18% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ± 53% +564.2% 0.10 ± 44% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown] 0.33 ± 33% -33.3% 0.22 ± 34% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.86 ± 27% -77.3% 0.20 perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 1.29 ± 27% -89.3% 0.14 ± 40% perf-sched.sch_delay.avg.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 0.02 ± 23% -56.7% 0.01 ± 8% perf-sched.sch_delay.avg.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 0.34 ± 15% -69.1% 0.10 ± 2% perf-sched.sch_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 9.19 ± 61% -99.8% 0.01 ± 67% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.02 ± 20% +287.4% 0.09 perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 17.13 ± 92% -99.0% 0.18 ±112% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +3485.6% 0.15 ±164% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.vma_alloc_folio_noprof 0.10 ±158% +1534.9% 1.65 ± 50% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 863.48 ± 36% -70.7% 252.62 ±149% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.00 ±223% +5.5e+05% 4.62 ± 63% perf-sched.sch_delay.max.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 1.28 ±217% +504.3% 7.74 ± 17% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.fdget_pos.ksys_write.do_syscall_64 203.65 ± 26% -95.4% 9.44 ± 10% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.04 ±223% +444.1% 0.20 ± 71% perf-sched.sch_delay.max.ms.__cond_resched.pipe_write.vfs_write.ksys_write.do_syscall_64 0.54 ±142% +1407.1% 8.13 ± 32% perf-sched.sch_delay.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.27 ±135% +1223.4% 3.63 ± 81% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown] 2014 ± 40% -99.0% 19.92 ± 29% perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 299.74 ± 13% -96.5% 10.45 ± 64% perf-sched.sch_delay.max.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 328.92 ± 16% -91.7% 27.21 ±166% perf-sched.sch_delay.max.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 309.06 ± 14% -89.4% 32.70 ±100% perf-sched.sch_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 444.84 ± 29% -99.7% 1.19 ±129% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 939.99 ± 61% -94.4% 52.33 ±127% perf-sched.sch_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1372 ± 80% -98.8% 16.87 ±160% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.03 ± 20% +127.0% 0.07 perf-sched.total_sch_delay.average.ms 2159 ± 40% -87.1% 278.03 ±131% perf-sched.total_sch_delay.max.ms 0.12 ± 15% +213.1% 0.38 ± 2% perf-sched.total_wait_and_delay.average.ms 12670156 ± 20% -65.4% 4386881 perf-sched.total_wait_and_delay.count.ms 5167 ± 12% -20.5% 4108 perf-sched.total_wait_and_delay.max.ms 0.09 ± 14% +242.6% 0.31 ± 2% perf-sched.total_wait_time.average.ms 38.76 ± 39% -69.3% 11.90 ± 15% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 2.00 ± 29% -68.8% 0.62 ± 15% perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.35 ±149% +330.4% 1.52 ± 45% perf-sched.wait_and_delay.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.66 ± 33% -38.7% 0.41 ± 53% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 2.94 ± 47% -81.1% 0.56 ± 62% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown] 3.20 ± 14% -51.5% 1.55 ± 23% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 2.99 ± 27% -86.5% 0.40 ± 29% perf-sched.wait_and_delay.avg.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 0.05 ± 22% +215.4% 0.17 perf-sched.wait_and_delay.avg.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 5.84 ± 22% -94.9% 0.30 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 22.31 ± 37% -86.9% 2.92 ± 15% perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 60.96 ± 64% -95.9% 2.48 ± 15% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 694.35 ± 24% -34.8% 452.48 perf-sched.wait_and_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 41.35 ± 51% -61.4% 15.94 ± 8% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 837.15 ± 9% -38.7% 513.26 ± 3% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.05 ± 18% +287.7% 0.19 perf-sched.wait_and_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 974.67 ± 43% -81.2% 183.00 ± 98% perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.folio_alloc_mpol_noprof.shmem_alloc_folio 172.83 ± 12% +115.7% 372.80 ± 6% perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 2935 ± 44% -60.9% 1147 ± 26% perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.00 ±141% +17520.0% 176.20 ± 25% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.fdget_pos.ksys_write.do_syscall_64 1742 ± 16% +66.8% 2906 ± 8% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 517.00 ± 55% +596.3% 3600 ± 23% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 1.00 ±141% +32440.0% 325.40 ± 19% perf-sched.wait_and_delay.count.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 885.67 ± 41% -65.1% 308.80 ± 54% perf-sched.wait_and_delay.count.__cond_resched.shmem_inode_acct_blocks.shmem_alloc_and_add_folio.shmem_get_folio_gfp.shmem_write_begin 1.17 ± 76% +328.6% 5.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 46.17 ± 21% +102.3% 93.40 ± 3% perf-sched.wait_and_delay.count.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.33 ± 61% +203.2% 19.20 ± 5% perf-sched.wait_and_delay.count.irq_thread.kthread.ret_from_fork.ret_from_fork_asm 827.50 ± 20% +192.3% 2418 ± 8% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 23590 ± 24% +194.3% 69429 ± 22% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 34754 ± 32% +467.3% 197159 ± 28% perf-sched.wait_and_delay.count.pipe_wait_readable.ipipe_prep.part.0.do_tee 6287013 ± 20% -74.8% 1585356 ± 3% perf-sched.wait_and_delay.count.pipe_wait_writable.opipe_prep.part.0.do_tee 46312 ± 21% +1467.4% 725907 ± 6% perf-sched.wait_and_delay.count.pipe_write.vfs_write.ksys_write.do_syscall_64 24.50 ± 22% +92.7% 47.20 ± 15% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 11.67 ± 24% +98.9% 23.20 perf-sched.wait_and_delay.count.schedule_hrtimeout_range.do_poll.constprop.0.do_sys_poll 40.67 ± 22% +114.4% 87.20 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 8.83 ± 28% +121.9% 19.60 ± 4% perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 319.00 ± 16% +103.7% 649.80 ± 3% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 6266766 ± 20% -71.4% 1792099 perf-sched.wait_and_delay.count.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 15.00 ± 27% +144.0% 36.60 perf-sched.wait_and_delay.count.wait_for_partner.fifo_open.do_dentry_open.vfs_open 207.17 ± 33% +104.6% 423.80 ± 8% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1843 ± 20% -34.8% 1201 ± 33% perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 2.56 ±218% +505.4% 15.49 ± 17% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.fdget_pos.ksys_write.do_syscall_64 407.31 ± 26% -95.4% 18.88 ± 10% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 1.04 ±149% +1457.6% 16.26 ± 32% perf-sched.wait_and_delay.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 1185 ± 57% -94.0% 70.68 ±122% perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 4839 ± 21% -63.0% 1788 ± 64% perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 599.48 ± 13% -94.8% 30.94 ± 68% perf-sched.wait_and_delay.max.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 657.85 ± 16% -91.5% 55.93 ±160% perf-sched.wait_and_delay.max.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 673.79 ± 21% -90.3% 65.42 ±100% perf-sched.wait_and_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 284.00 ± 52% -95.5% 12.82 ±110% perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 1366 ± 52% -98.7% 17.62 ± 86% perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 2100 ± 52% -76.0% 505.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 1044 ± 29% -61.4% 403.20 ± 22% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 2349 ± 19% -57.4% 1000 perf-sched.wait_and_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 25.60 ± 21% -56.9% 11.03 ± 5% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.00 ±223% +74228.0% 0.62 ± 53% perf-sched.wait_time.avg.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 1.02 ± 28% -67.7% 0.33 ± 13% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.18 ±140% +313.8% 0.76 ± 45% perf-sched.wait_time.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.33 ± 33% -33.3% 0.22 ± 34% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 2.63 ± 50% -87.7% 0.32 ± 47% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown] 2.34 ± 14% -42.1% 1.36 ± 26% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 1.70 ± 27% -84.4% 0.26 ± 24% perf-sched.wait_time.avg.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 0.03 ± 20% +408.2% 0.16 perf-sched.wait_time.avg.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 5.51 ± 23% -96.5% 0.19 ± 2% perf-sched.wait_time.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 19.69 ± 35% -85.8% 2.79 ± 8% perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +20831.4% 0.24 ± 55% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_double_lock 49.52 ± 60% -95.6% 2.19 ± 15% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 694.34 ± 24% -34.8% 452.47 perf-sched.wait_time.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 837.02 ± 9% -38.7% 513.25 ± 3% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.03 ± 17% +287.0% 0.10 perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.19 ±131% +1122.8% 2.33 ± 76% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 0.00 ±223% +5.5e+05% 4.62 ± 63% perf-sched.wait_time.max.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 1.28 ±217% +504.3% 7.74 ± 17% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.fdget_pos.ksys_write.do_syscall_64 203.65 ± 26% -95.4% 9.44 ± 10% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.54 ±142% +1407.1% 8.13 ± 32% perf-sched.wait_time.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 32.46 ± 99% -80.2% 6.42 ± 74% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown] 300.67 ± 13% -91.3% 26.05 ± 75% perf-sched.wait_time.max.ms.pipe_wait_readable.ipipe_prep.part.0.do_tee 328.99 ± 16% -86.2% 45.34 ±151% perf-sched.wait_time.max.ms.pipe_wait_writable.opipe_prep.part.0.do_tee 391.47 ± 23% -86.7% 51.90 ±136% perf-sched.wait_time.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 239.99 ± 46% -96.1% 9.32 ± 77% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +49648.6% 0.58 ± 61% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_double_lock 864.13 ± 37% -98.0% 17.07 ± 89% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 2100 ± 52% -76.0% 504.99 perf-sched.wait_time.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 2172 ± 17% -54.0% 1000 perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 16.97 -15.7 1.26 ± 3% perf-profile.calltrace.cycles-pp.tee 15.24 -14.7 0.59 ± 2% perf-profile.calltrace.cycles-pp.opipe_prep.do_tee.__x64_sys_tee.do_syscall_64.entry_SYSCALL_64_after_hwframe 15.19 -14.6 0.56 ± 3% perf-profile.calltrace.cycles-pp.pipe_wait_writable.opipe_prep.do_tee.__x64_sys_tee.do_syscall_64 15.29 -14.6 0.69 ± 3% perf-profile.calltrace.cycles-pp.do_tee.__x64_sys_tee.do_syscall_64.entry_SYSCALL_64_after_hwframe.tee 15.30 -14.6 0.73 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_tee.do_syscall_64.entry_SYSCALL_64_after_hwframe.tee 15.34 -14.4 0.94 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.tee 15.34 -14.4 0.96 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.tee 14.06 -14.0 0.10 ±200% perf-profile.calltrace.cycles-pp.schedule.pipe_wait_writable.opipe_prep.do_tee.__x64_sys_tee 13.67 -13.6 0.10 ±200% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait_writable.opipe_prep.do_tee 40.88 -11.1 29.80 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 10.86 -10.9 0.00 perf-profile.calltrace.cycles-pp.__wake_up_sync_key.pipe_read.vfs_read.ksys_read.do_syscall_64 13.96 -10.7 3.26 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 41.52 -10.6 30.89 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 10.50 -10.5 0.00 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.pipe_read.vfs_read.ksys_read 10.35 -10.3 0.00 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read.vfs_read 10.12 -10.1 0.00 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_read 9.38 -9.4 0.00 perf-profile.calltrace.cycles-pp.schedule.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 9.02 -9.0 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 52.99 -7.3 45.71 perf-profile.calltrace.cycles-pp.read 6.05 ± 2% -6.0 0.00 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key 5.49 -5.5 0.00 perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.pipe_wait_writable.opipe_prep 5.25 -5.2 0.00 perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.pipe_wait_writable 5.02 -5.0 0.00 perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule 20.87 -4.0 16.84 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 23.38 -2.3 21.12 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 25.16 -1.2 23.92 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.91 -0.2 0.73 ± 2% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read 0.54 ± 2% +0.3 0.83 ± 2% perf-profile.calltrace.cycles-pp.ktime_get_coarse_real_ts64_mg.current_time.atime_needs_update.touch_atime.pipe_read 0.61 +0.5 1.07 perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.72 +0.5 1.24 ± 2% perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.vfs_read.ksys_read.do_syscall_64 1.07 +0.5 1.62 ± 2% perf-profile.calltrace.cycles-pp.fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 0.78 +0.6 1.42 perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.vfs_write.ksys_write.do_syscall_64 0.00 +0.6 0.64 perf-profile.calltrace.cycles-pp.__cond_resched.mutex_lock.pipe_read.vfs_read.ksys_read 0.00 +0.7 0.66 perf-profile.calltrace.cycles-pp.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 0.17 ±141% +0.7 0.91 perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.7 0.75 perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.17 +0.8 1.93 perf-profile.calltrace.cycles-pp.current_time.atime_needs_update.touch_atime.pipe_read.vfs_read 0.00 +0.8 0.76 ± 4% perf-profile.calltrace.cycles-pp.stress_tee_pipe_write 0.00 +0.8 0.77 perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 0.08 ±223% +0.8 0.88 perf-profile.calltrace.cycles-pp.ktime_get_coarse_real_ts64_mg.current_time.inode_needs_update_time.file_update_time.pipe_write 0.00 +0.8 0.80 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write 1.14 +0.8 1.96 ± 2% perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.vfs_read.ksys_read.do_syscall_64 0.98 +0.9 1.84 ± 2% perf-profile.calltrace.cycles-pp.fdget_pos.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 1.16 +0.9 2.08 perf-profile.calltrace.cycles-pp.current_time.inode_needs_update_time.file_update_time.pipe_write.vfs_write 1.18 +1.0 2.13 perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.vfs_write.ksys_write.do_syscall_64 1.63 +1.3 2.93 perf-profile.calltrace.cycles-pp.inode_needs_update_time.file_update_time.pipe_write.vfs_write.ksys_write 2.33 +1.4 3.72 perf-profile.calltrace.cycles-pp.atime_needs_update.touch_atime.pipe_read.vfs_read.ksys_read 1.72 +1.4 3.16 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 2.63 +1.6 4.22 perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.vfs_read.ksys_read.do_syscall_64 1.98 +1.6 3.57 perf-profile.calltrace.cycles-pp.file_update_time.pipe_write.vfs_write.ksys_write.do_syscall_64 2.86 +2.1 4.95 perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.pipe_read.vfs_read.ksys_read 3.43 +2.5 5.92 perf-profile.calltrace.cycles-pp.copy_page_to_iter.pipe_read.vfs_read.ksys_read.do_syscall_64 2.99 +2.6 5.54 perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.pipe_write.vfs_write.ksys_write 3.52 +2.6 6.11 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read 4.25 +2.9 7.12 perf-profile.calltrace.cycles-pp.clear_bhb_loop.read 3.50 +3.0 6.47 perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.vfs_write.ksys_write.do_syscall_64 3.75 +3.2 6.93 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write 4.49 +3.8 8.33 perf-profile.calltrace.cycles-pp.clear_bhb_loop.write 9.78 +8.2 17.98 perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.20 +10.2 22.45 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 13.89 +11.7 25.59 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 17.18 +14.4 31.62 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 17.86 +15.0 32.86 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 26.99 +22.8 49.78 perf-profile.calltrace.cycles-pp.write 23.67 -22.6 1.11 ± 2% perf-profile.children.cycles-pp.schedule 23.19 -22.1 1.09 ± 2% perf-profile.children.cycles-pp.__schedule 16.98 -15.7 1.30 ± 3% perf-profile.children.cycles-pp.tee 15.24 -14.7 0.56 ± 3% perf-profile.children.cycles-pp.pipe_wait_writable 15.24 -14.7 0.59 ± 3% perf-profile.children.cycles-pp.opipe_prep 15.29 -14.6 0.70 ± 3% perf-profile.children.cycles-pp.do_tee 15.30 -14.6 0.73 ± 3% perf-profile.children.cycles-pp.__x64_sys_tee 73.87 -10.7 63.20 perf-profile.children.cycles-pp.do_syscall_64 10.91 -10.5 0.45 ± 5% perf-profile.children.cycles-pp.__wake_up_sync_key 10.61 -10.1 0.52 ± 7% perf-profile.children.cycles-pp.__wake_up_common 75.40 -10.0 65.40 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 10.44 -9.9 0.51 ± 6% perf-profile.children.cycles-pp.autoremove_wake_function 10.32 -9.8 0.51 ± 6% perf-profile.children.cycles-pp.try_to_wake_up 16.00 -9.0 6.96 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 53.19 -7.3 45.93 perf-profile.children.cycles-pp.read 6.34 -6.1 0.29 ± 2% perf-profile.children.cycles-pp.__pick_next_task 6.16 -5.9 0.28 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 6.13 ± 2% -5.8 0.28 perf-profile.children.cycles-pp.ttwu_do_activate 5.56 -5.3 0.27 ± 4% perf-profile.children.cycles-pp.try_to_block_task 5.32 -5.1 0.26 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 5.12 -4.9 0.25 ± 3% perf-profile.children.cycles-pp.dequeue_entities 4.92 ± 3% -4.7 0.25 ± 2% perf-profile.children.cycles-pp.enqueue_task 4.69 ± 3% -4.5 0.23 ± 2% perf-profile.children.cycles-pp.enqueue_task_fair 4.57 -4.3 0.22 ± 5% perf-profile.children.cycles-pp.update_curr 4.42 -4.2 0.21 perf-profile.children.cycles-pp.update_load_avg 4.31 -4.1 0.19 ± 2% perf-profile.children.cycles-pp.switch_mm_irqs_off 21.23 -3.8 17.41 perf-profile.children.cycles-pp.pipe_read 3.28 -3.1 0.15 ± 6% perf-profile.children.cycles-pp.dequeue_entity 2.87 ± 3% -2.7 0.12 ± 3% perf-profile.children.cycles-pp.enqueue_entity 23.53 -2.2 21.36 perf-profile.children.cycles-pp.vfs_read 2.17 -2.1 0.09 ± 4% perf-profile.children.cycles-pp.pick_task_fair 1.90 -1.8 0.09 ± 7% perf-profile.children.cycles-pp.set_next_entity 1.70 -1.6 0.08 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se 1.70 -1.6 0.09 ± 4% perf-profile.children.cycles-pp.prepare_task_switch 1.59 -1.5 0.07 ± 7% perf-profile.children.cycles-pp.__switch_to_asm 1.65 -1.5 0.14 perf-profile.children.cycles-pp.__rseq_handle_notify_resume 1.54 -1.5 0.07 perf-profile.children.cycles-pp.__switch_to 1.43 -1.4 0.07 perf-profile.children.cycles-pp.put_prev_entity 1.39 -1.3 0.06 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.36 -1.3 0.07 perf-profile.children.cycles-pp.wakeup_preempt 1.28 -1.2 0.07 perf-profile.children.cycles-pp.pick_eevdf 25.30 -1.1 24.16 perf-profile.children.cycles-pp.ksys_read 1.17 ± 9% -1.1 0.06 ± 8% perf-profile.children.cycles-pp.select_task_rq 0.98 ± 11% -0.9 0.04 ± 50% perf-profile.children.cycles-pp.select_task_rq_fair 0.99 -0.9 0.06 ± 6% perf-profile.children.cycles-pp.check_preempt_wakeup_fair 0.90 -0.8 0.08 ± 6% perf-profile.children.cycles-pp.rseq_ip_fixup 1.05 ± 16% -0.7 0.34 ±113% perf-profile.children.cycles-pp.reader__read_event 1.05 ± 16% -0.7 0.34 ±112% perf-profile.children.cycles-pp.perf_session__process_events 1.05 ± 16% -0.7 0.34 ±112% perf-profile.children.cycles-pp.record__finish_output 0.71 -0.6 0.06 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.61 -0.6 0.04 ± 50% perf-profile.children.cycles-pp.os_xsave 0.62 ± 5% -0.5 0.08 ± 5% perf-profile.children.cycles-pp.reweight_entity 0.59 -0.5 0.05 ± 7% perf-profile.children.cycles-pp.prepare_to_wait_event 0.22 ± 2% -0.1 0.11 perf-profile.children.cycles-pp.switch_fpu_return 0.19 ± 3% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.16 ± 2% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.update_process_times 0.18 ± 4% -0.0 0.16 ± 4% perf-profile.children.cycles-pp.tick_nohz_handler 0.08 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.sched_tick 0.07 +0.0 0.09 ± 15% perf-profile.children.cycles-pp.__wake_up 0.09 ± 5% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.page_counter_cancel 0.09 ± 10% +0.0 0.13 ± 9% perf-profile.children.cycles-pp.page_counter_uncharge 0.07 ± 5% +0.0 0.12 ± 10% perf-profile.children.cycles-pp.__splice_from_pipe 0.11 ± 9% +0.1 0.16 ± 9% perf-profile.children.cycles-pp.uncharge_batch 0.07 ± 6% +0.1 0.13 ± 5% perf-profile.children.cycles-pp.__alloc_pages_noprof 0.08 ± 5% +0.1 0.14 ± 7% perf-profile.children.cycles-pp.alloc_pages_mpol_noprof 0.12 ± 11% +0.1 0.18 ± 9% perf-profile.children.cycles-pp.__mem_cgroup_uncharge 0.12 ± 9% +0.1 0.18 ± 8% perf-profile.children.cycles-pp.__folio_put 0.00 +0.1 0.06 ± 12% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.08 +0.1 0.15 ± 7% perf-profile.children.cycles-pp.splice_from_pipe 0.18 ± 8% +0.1 0.26 ± 2% perf-profile.children.cycles-pp.__x64_sys_read 0.00 +0.1 0.09 perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 0.09 ± 5% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.do_splice 0.09 ± 4% +0.1 0.22 ± 6% perf-profile.children.cycles-pp.__x64_sys_splice 0.19 +0.1 0.32 ± 2% perf-profile.children.cycles-pp.make_vfsuid 0.19 +0.1 0.32 ± 3% perf-profile.children.cycles-pp.make_vfsgid 0.16 ± 2% +0.1 0.29 perf-profile.children.cycles-pp.__x64_sys_write 0.22 ± 5% +0.1 0.36 ± 3% perf-profile.children.cycles-pp.read@plt 0.19 ± 2% +0.2 0.34 ± 6% perf-profile.children.cycles-pp.write@plt 1.00 +0.2 1.16 ± 3% perf-profile.children.cycles-pp.stress_tee_pipe_read 0.14 ± 3% +0.2 0.33 ± 5% perf-profile.children.cycles-pp.splice 0.36 +0.2 0.60 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.38 +0.3 0.68 perf-profile.children.cycles-pp.kill_fasync 1.48 +0.3 1.78 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.40 +0.3 0.73 perf-profile.children.cycles-pp.rcu_all_qs 0.56 +0.4 0.96 perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.56 +0.4 1.00 perf-profile.children.cycles-pp.stress_tee_pipe_write 0.58 +0.5 1.04 perf-profile.children.cycles-pp.security_file_permission 1.13 +0.7 1.88 perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg 0.96 +0.8 1.72 perf-profile.children.cycles-pp.x64_sys_call 0.92 +0.8 1.69 perf-profile.children.cycles-pp.__cond_resched 1.30 +1.0 2.33 perf-profile.children.cycles-pp.rw_verify_area 1.59 +1.3 2.87 perf-profile.children.cycles-pp.mutex_unlock 1.72 +1.4 3.11 perf-profile.children.cycles-pp.inode_needs_update_time 2.14 +1.5 3.62 perf-profile.children.cycles-pp.fdget_pos 2.51 +1.5 4.04 perf-profile.children.cycles-pp.atime_needs_update 2.72 +1.7 4.37 perf-profile.children.cycles-pp.touch_atime 2.08 +1.7 3.75 perf-profile.children.cycles-pp.file_update_time 2.52 +1.8 4.35 perf-profile.children.cycles-pp.current_time 2.52 +2.0 4.49 perf-profile.children.cycles-pp.mutex_lock 2.92 +2.1 5.03 perf-profile.children.cycles-pp._copy_to_iter 3.53 +2.6 6.09 perf-profile.children.cycles-pp.copy_page_to_iter 3.04 +2.6 5.64 perf-profile.children.cycles-pp._copy_from_iter 3.27 +2.7 5.96 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 3.65 +3.1 6.75 perf-profile.children.cycles-pp.copy_page_from_iter 4.17 +3.3 7.49 perf-profile.children.cycles-pp.entry_SYSCALL_64 8.86 +6.9 15.76 perf-profile.children.cycles-pp.clear_bhb_loop 10.12 +8.5 18.62 perf-profile.children.cycles-pp.pipe_write 12.38 +10.4 22.74 perf-profile.children.cycles-pp.vfs_write 14.06 +11.8 25.89 perf-profile.children.cycles-pp.ksys_write 27.17 +22.9 50.09 perf-profile.children.cycles-pp.write 4.22 -4.0 0.19 perf-profile.self.cycles-pp.switch_mm_irqs_off 2.34 -2.2 0.11 ± 4% perf-profile.self.cycles-pp.__schedule 1.73 ± 4% -1.6 0.08 ± 12% perf-profile.self.cycles-pp.update_curr 1.57 -1.5 0.07 ± 7% perf-profile.self.cycles-pp.__switch_to_asm 1.56 -1.5 0.08 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se 1.46 -1.4 0.07 ± 7% perf-profile.self.cycles-pp.__switch_to 1.43 -1.4 0.07 perf-profile.self.cycles-pp.update_load_avg 1.30 -1.2 0.06 ± 6% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.82 ± 2% -0.8 0.05 ± 7% perf-profile.self.cycles-pp.prepare_task_switch 0.81 -0.8 0.04 ± 50% perf-profile.self.cycles-pp.pick_eevdf 0.67 -0.6 0.05 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.60 -0.6 0.04 ± 50% perf-profile.self.cycles-pp.os_xsave 0.08 ± 12% +0.0 0.10 ± 6% perf-profile.self.cycles-pp.read@plt 0.08 ± 5% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.page_counter_cancel 0.13 ± 11% +0.0 0.18 ± 4% perf-profile.self.cycles-pp.__x64_sys_read 0.05 +0.1 0.10 ± 17% perf-profile.self.cycles-pp.write@plt 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.tee 0.11 ± 3% +0.1 0.19 perf-profile.self.cycles-pp.__x64_sys_write 0.00 +0.1 0.09 perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 0.14 ± 2% +0.1 0.24 ± 4% perf-profile.self.cycles-pp.make_vfsgid 0.14 +0.1 0.24 perf-profile.self.cycles-pp.make_vfsuid 0.20 ± 2% +0.1 0.32 perf-profile.self.cycles-pp.touch_atime 0.20 ± 2% +0.1 0.34 ± 2% perf-profile.self.cycles-pp.kill_fasync 0.87 +0.2 1.04 ± 3% perf-profile.self.cycles-pp.stress_tee_pipe_read 0.30 +0.2 0.54 perf-profile.self.cycles-pp.rcu_all_qs 0.34 +0.2 0.59 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.35 +0.3 0.63 perf-profile.self.cycles-pp.file_update_time 0.45 +0.3 0.78 perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.50 ± 2% +0.4 0.86 ± 2% perf-profile.self.cycles-pp.stress_tee_pipe_write 0.47 +0.4 0.85 perf-profile.self.cycles-pp.inode_needs_update_time 0.48 +0.4 0.86 perf-profile.self.cycles-pp.security_file_permission 0.62 +0.4 1.06 perf-profile.self.cycles-pp.copy_page_to_iter 0.51 +0.4 0.96 perf-profile.self.cycles-pp.__cond_resched 0.70 +0.5 1.18 ± 2% perf-profile.self.cycles-pp.ksys_read 0.96 +0.5 1.47 perf-profile.self.cycles-pp.atime_needs_update 0.66 +0.5 1.19 perf-profile.self.cycles-pp.copy_page_from_iter 1.18 +0.6 1.74 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.72 +0.6 1.29 perf-profile.self.cycles-pp.rw_verify_area 0.71 +0.6 1.31 perf-profile.self.cycles-pp.ksys_write 1.51 +0.7 2.18 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.85 +0.7 1.52 perf-profile.self.cycles-pp.x64_sys_call 1.03 +0.7 1.70 perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg 1.04 +0.8 1.82 perf-profile.self.cycles-pp.entry_SYSCALL_64 1.87 +1.0 2.92 perf-profile.self.cycles-pp.pipe_read 1.39 +1.1 2.47 perf-profile.self.cycles-pp.current_time 1.64 +1.1 2.78 ± 2% perf-profile.self.cycles-pp.vfs_read 1.59 +1.2 2.78 perf-profile.self.cycles-pp.mutex_lock 1.49 +1.2 2.68 perf-profile.self.cycles-pp.mutex_unlock 1.81 +1.2 3.04 perf-profile.self.cycles-pp.read 2.03 +1.4 3.39 perf-profile.self.cycles-pp.do_syscall_64 2.03 +1.4 3.44 ± 2% perf-profile.self.cycles-pp.fdget_pos 1.64 +1.4 3.07 perf-profile.self.cycles-pp.vfs_write 1.84 +1.6 3.43 perf-profile.self.cycles-pp.write 3.60 +1.8 5.36 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 2.13 +1.8 3.94 ± 2% perf-profile.self.cycles-pp.pipe_write 2.86 +2.1 4.93 perf-profile.self.cycles-pp._copy_to_iter 2.93 +2.5 5.44 perf-profile.self.cycles-pp._copy_from_iter 3.16 +2.6 5.75 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 8.75 +6.8 15.58 perf-profile.self.cycles-pp.clear_bhb_loop *************************************************************************************************** lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-12/performance/pipe/4/x86_64-rhel-9.4/process/50%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp2/hackbench commit: d2fc0ed52a ("Merge branch 'vfs-6.14.uncached_buffered_io'") aaec5a95d5 ("pipe_read: don't wake up the writer if the pipe is still full") d2fc0ed52a284a13 aaec5a95d59615523db03dd53c2 ---------------- --------------------------- %stddev %change %stddev \ | \ 0.04 ± 2% -0.0 0.03 ± 6% mpstat.cpu.all.soft% 13470 ± 44% +47.4% 19860 ± 22% numa-vmstat.node1.nr_slab_reclaimable 53825 ± 44% +47.6% 79444 ± 22% numa-meminfo.node1.KReclaimable 53825 ± 44% +47.6% 79444 ± 22% numa-meminfo.node1.SReclaimable 1423 -14.8% 1212 ± 2% vmstat.procs.r 1419871 ± 3% +22.4% 1737760 ± 2% vmstat.system.in 28551 ± 13% -37.8% 17760 ± 8% perf-c2c.DRAM.remote 122888 ± 7% +55.5% 191041 ± 2% perf-c2c.HITM.local 4310 ± 13% -20.0% 3448 ± 9% perf-c2c.HITM.remote 127198 ± 7% +52.9% 194489 ± 2% perf-c2c.HITM.total 903976 -7.5% 836425 hackbench.throughput 866824 -7.6% 801128 hackbench.throughput_avg 903976 -7.5% 836425 hackbench.throughput_best 785145 -6.8% 731474 hackbench.throughput_worst 69.85 +8.1% 75.53 hackbench.time.elapsed_time 69.85 +8.1% 75.53 hackbench.time.elapsed_time.max 1.986e+08 +22.1% 2.424e+08 hackbench.time.involuntary_context_switches 7559 +9.7% 8296 hackbench.time.system_time 985.62 +4.1% 1026 hackbench.time.user_time 6.06 ± 4% -16.2% 5.08 ± 2% sched_debug.cfs_rq:/.h_nr_running.avg 14.20 ± 13% -19.0% 11.50 ± 4% sched_debug.cfs_rq:/.h_nr_running.max 3.32 ± 3% -18.4% 2.71 ± 2% sched_debug.cfs_rq:/.h_nr_running.stddev 5996 ± 2% +34.3% 8054 ± 23% sched_debug.cfs_rq:/.load.avg 24211 ± 8% +1091.3% 288431 ± 88% sched_debug.cfs_rq:/.load.max 4439 ± 3% +504.9% 26851 ± 81% sched_debug.cfs_rq:/.load.stddev 1.00 ± 54% +83.3% 1.83 ± 12% sched_debug.cfs_rq:/.load_avg.min 6203 ± 2% -15.4% 5246 ± 2% sched_debug.cfs_rq:/.runnable_avg.avg 1886 ± 7% -22.5% 1461 ± 7% sched_debug.cfs_rq:/.runnable_avg.stddev 151.50 ± 28% +55.7% 235.83 ± 11% sched_debug.cfs_rq:/.util_avg.min 667.32 ± 3% -21.3% 525.24 ± 8% sched_debug.cfs_rq:/.util_est.avg 6.06 ± 4% -16.1% 5.09 ± 2% sched_debug.cpu.nr_running.avg 14.20 ± 13% -19.0% 11.50 ± 4% sched_debug.cpu.nr_running.max 3.30 ± 4% -18.7% 2.68 ± 2% sched_debug.cpu.nr_running.stddev 130727 ± 33% -46.7% 69684 ± 27% sched_debug.cpu.nr_switches.stddev 6.98 ± 11% -2.4 4.61 ± 24% perf-profile.calltrace.cycles-pp.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64 6.98 ± 11% -2.4 4.61 ± 24% perf-profile.calltrace.cycles-pp.shmem_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.67 ± 11% -2.1 4.61 ± 24% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn 6.67 ± 11% -2.1 4.61 ± 24% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write.writen.record__pushfn.perf_mmap__push 6.67 ± 11% -2.1 4.61 ± 24% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write.writen 3.02 ± 44% -1.9 1.09 ± 47% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.vfs_statx.vfs_fstatat 6.67 ± 11% -1.8 4.86 ± 17% perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record 6.67 ± 11% -1.8 4.86 ± 17% perf-profile.calltrace.cycles-pp.write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist 6.67 ± 11% -1.8 4.86 ± 17% perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record 2.72 ± 21% -1.2 1.50 ± 57% perf-profile.calltrace.cycles-pp.fault_in_iov_iter_readable.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.17 ±122% +2.5 3.65 ± 43% perf-profile.calltrace.cycles-pp.do_pte_missing.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 1.81 ± 62% +2.8 4.59 ± 21% perf-profile.calltrace.cycles-pp.link_path_walk.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 1.52 ± 65% +3.1 4.63 ± 39% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.52 ± 65% +3.1 4.63 ± 39% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.98 ± 11% -2.4 4.61 ± 24% perf-profile.children.cycles-pp.generic_perform_write 6.98 ± 11% -2.4 4.61 ± 24% perf-profile.children.cycles-pp.shmem_file_write_iter 3.93 ± 39% -2.2 1.75 ± 66% perf-profile.children.cycles-pp.mutex_unlock 6.67 ± 11% -1.8 4.86 ± 17% perf-profile.children.cycles-pp.record__pushfn 6.67 ± 11% -1.4 5.30 ± 11% perf-profile.children.cycles-pp.writen 2.72 ± 21% -1.2 1.50 ± 57% perf-profile.children.cycles-pp.fault_in_iov_iter_readable 2.72 ± 21% -1.2 1.50 ± 57% perf-profile.children.cycles-pp.fault_in_readable 1.17 ±122% +2.5 3.65 ± 43% perf-profile.children.cycles-pp.do_pte_missing 6.169e+10 -6.4% 5.773e+10 perf-stat.i.branch-instructions 2.704e+08 -3.6% 2.607e+08 perf-stat.i.branch-misses 13.54 ± 10% -4.4 9.18 ± 2% perf-stat.i.cache-miss-rate% 1.343e+09 ± 9% +40.2% 1.883e+09 ± 3% perf-stat.i.cache-references 12981250 ± 2% -2.9% 12601747 perf-stat.i.context-switches 1.18 +7.0% 1.26 perf-stat.i.cpi 656622 ± 3% +43.2% 940118 perf-stat.i.cpu-migrations 2.674e+11 -6.3% 2.506e+11 perf-stat.i.instructions 0.85 -6.3% 0.80 perf-stat.i.ipc 13648 ± 2% -7.1% 12676 ± 3% perf-stat.i.minor-faults 13650 ± 2% -7.1% 12677 ± 3% perf-stat.i.page-faults 0.44 +0.0 0.45 perf-stat.overall.branch-miss-rate% 13.19 ± 10% -4.3 8.94 ± 3% perf-stat.overall.cache-miss-rate% 1.18 +7.3% 1.26 perf-stat.overall.cpi 0.85 -6.8% 0.79 perf-stat.overall.ipc 6.089e+10 -6.3% 5.703e+10 perf-stat.ps.branch-instructions 2.666e+08 -3.5% 2.572e+08 perf-stat.ps.branch-misses 1.324e+09 ± 9% +40.3% 1.858e+09 ± 3% perf-stat.ps.cache-references 12791040 ± 2% -2.8% 12432936 perf-stat.ps.context-switches 646715 ± 3% +43.3% 926916 perf-stat.ps.cpu-migrations 2.64e+11 -6.2% 2.476e+11 perf-stat.ps.instructions 13461 ± 2% -7.7% 12426 ± 3% perf-stat.ps.minor-faults 13463 ± 2% -7.7% 12427 ± 3% perf-stat.ps.page-faults 3.22 ± 92% +235.4% 10.78 ± 42% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 0.24 ±100% -84.4% 0.04 ±102% perf-sched.sch_delay.avg.ms.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region 0.08 ± 90% -88.4% 0.01 ±112% perf-sched.sch_delay.avg.ms.__cond_resched.__tlb_batch_free_encoded_pages.tlb_finish_mmu.relocate_vma_down.setup_arg_pages 10.19 ±196% -99.8% 0.02 ±134% perf-sched.sch_delay.avg.ms.__cond_resched.change_pud_range.isra.0.change_protection_range 2.47 ±196% -99.8% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.open_last_lookups.path_openat.do_filp_open 0.10 ± 45% -83.6% 0.02 ±153% perf-sched.sch_delay.avg.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 0.09 ± 86% -92.9% 0.01 ±172% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma 7.03 ± 19% +243.1% 24.13 ± 13% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 5.62 ±195% -99.6% 0.02 ± 92% perf-sched.sch_delay.avg.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 15.78 ± 28% +338.0% 69.12 ± 19% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 0.99 ± 19% +59.4% 1.57 ± 12% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 5.84 ± 20% +945.2% 61.00 ± 11% perf-sched.sch_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 15.78 ± 21% +208.2% 48.63 ± 11% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 12.67 ± 71% +452.9% 70.03 ± 81% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1.22 ± 17% +252.3% 4.30 ± 11% perf-sched.sch_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.12 ± 83% +577.2% 0.83 ± 61% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open 0.20 ±104% -94.9% 0.01 ±115% perf-sched.sch_delay.max.ms.__cond_resched.__tlb_batch_free_encoded_pages.tlb_finish_mmu.relocate_vma_down.setup_arg_pages 20.36 ±196% -99.9% 0.02 ±129% perf-sched.sch_delay.max.ms.__cond_resched.change_pud_range.isra.0.change_protection_range 2.48 ±195% -99.7% 0.01 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_read.open_last_lookups.path_openat.do_filp_open 0.28 ± 88% -89.8% 0.03 ±148% perf-sched.sch_delay.max.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 0.14 ±105% -95.5% 0.01 ±172% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma 1095 ± 14% +118.2% 2389 ± 15% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 16.66 ±198% -99.7% 0.05 ±104% perf-sched.sch_delay.max.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 797.45 ± 24% +149.6% 1990 ± 17% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 1768 ± 22% +94.8% 3445 ± 11% perf-sched.sch_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 746.96 ± 15% +265.4% 2729 ± 21% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 0.01 ± 15% -33.9% 0.01 ± 45% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 1716 ± 29% +121.7% 3804 ± 17% perf-sched.sch_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.54 ±107% +338.8% 2.37 ± 51% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open 1.41 ± 19% +78.0% 2.51 ± 11% perf-sched.total_sch_delay.average.ms 2305 ± 20% +82.3% 4203 ± 11% perf-sched.total_sch_delay.max.ms 4.38 ± 18% +68.2% 7.37 ± 11% perf-sched.total_wait_and_delay.average.ms 4583511 ± 23% -32.5% 3091848 ± 17% perf-sched.total_wait_and_delay.count.ms 5607 ± 10% +48.2% 8309 ± 8% perf-sched.total_wait_and_delay.max.ms 2.97 ± 18% +63.5% 4.86 ± 11% perf-sched.total_wait_time.average.ms 5092 ± 4% +13.9% 5800 ± 6% perf-sched.total_wait_time.max.ms 11.10 ± 59% +210.8% 34.49 ± 24% perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 21.29 ± 17% +235.3% 71.40 ± 12% perf-sched.wait_and_delay.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 110.43 ± 47% +250.5% 387.09 ± 45% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 48.98 ± 24% +319.4% 205.43 ± 16% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 2.87 ± 19% +56.2% 4.48 ± 11% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 19.17 ± 20% +835.0% 179.22 ± 10% perf-sched.wait_and_delay.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 48.33 ± 24% +209.7% 149.70 ± 11% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 38.33 ± 45% +401.4% 192.16 ± 68% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 3.58 ± 17% +253.9% 12.65 ± 11% perf-sched.wait_and_delay.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 56.40 ± 39% -72.8% 15.33 ± 46% perf-sched.wait_and_delay.count.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 785.60 ± 53% -49.5% 396.83 ± 49% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown] 356336 ± 23% -93.6% 22658 ± 16% perf-sched.wait_and_delay.count.pipe_write.vfs_write.ksys_write.do_syscall_64 203.40 ± 27% -71.1% 58.83 ± 63% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 779.32 ± 74% +132.5% 1812 ± 26% perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 2209 ± 14% +116.3% 4779 ± 15% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 1679 ± 22% +139.3% 4019 ± 18% perf-sched.wait_and_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 3543 ± 22% +98.8% 7042 ± 10% perf-sched.wait_and_delay.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 1595 ± 26% +243.1% 5471 ± 21% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 5048 ± 4% +12.6% 5685 ± 6% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 4052 ± 22% +88.2% 7626 ± 17% perf-sched.wait_and_delay.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 7.88 ± 48% +200.8% 23.70 ± 26% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 0.24 ±100% -84.4% 0.04 ±102% perf-sched.wait_time.avg.ms.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region 6.22 ± 65% +237.5% 21.00 ± 59% perf-sched.wait_time.avg.ms.__cond_resched.__mutex_lock.constprop.0.pipe_write 0.08 ± 90% -88.4% 0.01 ±112% perf-sched.wait_time.avg.ms.__cond_resched.__tlb_batch_free_encoded_pages.tlb_finish_mmu.relocate_vma_down.setup_arg_pages 18.06 ± 14% +43.8% 25.98 ± 15% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 10.19 ±196% -99.8% 0.02 ±134% perf-sched.wait_time.avg.ms.__cond_resched.change_pud_range.isra.0.change_protection_range 2.47 ±196% -99.8% 0.00 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_read.open_last_lookups.path_openat.do_filp_open 0.10 ± 45% -83.5% 0.02 ±153% perf-sched.wait_time.avg.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 0.09 ± 86% -92.9% 0.01 ±172% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma 14.26 ± 16% +231.5% 47.27 ± 12% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 6.86 ±196% -99.6% 0.02 ± 92% perf-sched.wait_time.avg.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 2.44 ± 44% +137.1% 5.78 ± 46% perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.20 ± 23% +310.5% 136.31 ± 16% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 1.88 ± 19% +54.5% 2.91 ± 11% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 13.33 ± 21% +786.7% 118.22 ± 10% perf-sched.wait_time.avg.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 32.55 ± 25% +210.5% 101.07 ± 12% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 25.66 ± 34% +375.9% 122.14 ± 60% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 2.35 ± 18% +254.7% 8.35 ± 11% perf-sched.wait_time.avg.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.08 ± 84% +731.0% 0.70 ± 43% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open 502.86 ± 46% +115.0% 1081 ± 38% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_noprof.alloc_pages_mpol_noprof.pipe_write.vfs_write 188.34 ±175% -97.8% 4.23 ± 49% perf-sched.wait_time.max.ms.__cond_resched.__mutex_lock.constprop.0.pipe_read 0.20 ±104% -94.9% 0.01 ±115% perf-sched.wait_time.max.ms.__cond_resched.__tlb_batch_free_encoded_pages.tlb_finish_mmu.relocate_vma_down.setup_arg_pages 20.36 ±196% -99.9% 0.02 ±129% perf-sched.wait_time.max.ms.__cond_resched.change_pud_range.isra.0.change_protection_range 2.48 ±195% -99.7% 0.01 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_read.open_last_lookups.path_openat.do_filp_open 0.28 ± 88% -89.8% 0.03 ±148% perf-sched.wait_time.max.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 0.14 ±105% -95.5% 0.01 ±172% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma 1126 ± 15% +132.8% 2622 ± 13% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.pipe_write.vfs_write.ksys_write 20.38 ±198% -99.8% 0.05 ±104% perf-sched.wait_time.max.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 951.43 ± 18% +142.5% 2307 ± 15% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown] 1820 ± 24% +107.0% 3767 ± 7% perf-sched.wait_time.max.ms.pipe_write.vfs_write.ksys_write.do_syscall_64 994.33 ± 39% +179.3% 2776 ± 19% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.pipe_write 921.59 ± 39% +70.0% 1566 ± 29% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 5048 ± 4% +12.6% 5685 ± 6% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 2400 ± 36% +64.8% 3954 ± 12% perf-sched.wait_time.max.ms.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.42 ±103% +470.3% 2.37 ± 50% perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
