On Tue, May 20, 2025 at 4:35 AM John Ogness <john.ogness@xxxxxxxxxxxxx> wrote: > > On 2025-05-20, Leonardo Bras <leobras@xxxxxxxxxx> wrote: > > When running cyclictest with break trace (-b) option, wait for skip_sec > > seconds before issuing the break. > > > > Cyclictest may present high latency on the initial cycles, which can be > > caused by initial resource allocations, and may not represent the > > expected latency for the running workload. > > If cyclictest is programmed correctly, this will not happen. Can you > provide a technical explanation for high latencies on initial cycles? Hi John, We are currently investigating the source of this latency, but I heard from other team members it's also happening on Intel Secure VMs as well. Scenario: Host: ARM64, kernel-rt, 120 out of 128 cpu isolated, hugepages for guest KVM Guest: kernel-rt, 120 vcpus pinned on above isolated cpus, 116 vcpus isolated on guest Cyclictest runs with trace-cmd attaching to a guest agent: trace-cmd record --poll -m 1000 -e csd -e sched/sched_switch -e timer -e irq_handler_entry -e irq_handler_exit -e tick_stop -e ipi_send_cpumask -e ipi_send_cpu -A 3 -e csd -e sched/sched_switch -e timer -e irq_handler_entry -e irq_handler_exit -e tick_stop -e ipi_send_cpumask -e ipi_send_cpu bash -c "ssh $my_guest 'cyclictest -m -q -p95 --policy=fifo -D 2h -i 200 -h60 -t 116 -a 4-119 -b 50 --mainaffinity 0,1,2,3 --tracemark'" What we see is a peak of >50us in the first couple cycles, and then a max peak of 15 in our tests. We also tested a sleep before the above cyclictest command, in order to remove any trace-cmd setup that might have been in progress, but it did not help. If you have any idea on why this might happen, or suggest a debugging setup, it would be appreciated. Thanks! Leo
