[1.] One line summary: [KVM/VMX] Level triggered interrupts mishandled on Windows w/ nested virt(Credential Guard) when using split irqchip [2.] Problem/Report: When running Windows with Credential Guard enabled and with split-irqchip, level triggered interrupts are not properly forwarded to L2 (Credential Guard) by L1 (Windows), instead L1 EOIs the interrupt. Which leads to extremely slow Windows boot time. This issue is only seen on Intel + split-irqchip. Intel + kernel-irqchip, AMD + (kernel/split)-irqchip works fine. Qemu command used to create the vm: /usr/libexec/qemu-kvm \ -machine q35,accel=kvm,smm=on,usb=off,acpi=on,kernel-irqchip=split \ -cpu host,+vmx,+invpcid,+ssse3,+aes,+xsave,+xsaveopt,+xgetbv1,+xsaves,+rdtscp,+tsc-deadline \ -m 20G -smp 1 \ -boot order=c,menu=on \ -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE_4MB.secboot.fd \ -drive if=pflash,format=raw,file=/var/lib/qemu/nvram/win_VARS.fd \ -chardev socket,id=chrtpm,path=/var/lib/qemu/tpm-win/sock \ -tpmdev emulator,id=tpm0,chardev=chrtpm \ -device tpm-crb,tpmdev=tpm0 \ -device ich9-usb-ehci1,id=usb \ -device ich9-usb-uhci1,masterbus=usb.0,firstport=0 \ -device ich9-usb-uhci2,masterbus=usb.0,firstport=2 \ -device ich9-usb-uhci3,masterbus=usb.0,firstport=4 \ -device usb-tablet,bus=usb.0,port=1 \ -device VGA,vgamem_mb=16 \ -device ich9-ahci,id=sata \ -drive file=/var/log/winsrv_2022_cg.qcow2,if=none,format=qcow2,id=hd0 \ -device ide-hd,drive=hd0,bus=sata.0 \ -vnc 0.0.0.0:5 \ -serial mon:stdio Note that usb controllers and usb devices generate the INTx interrupts hence are important for repro, the qcow2 is Windows Server 2022 with UEFI boot and Credential Guard. Removing kernel-irqchip=split (using kernel-irqchip) does not lead into the issue. Processor Information: Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz Kernel Info: 6.12.26-11.0s20c78r5.el9.x86_64 (bug present) 6.17.0-11.0s20c79r5.el9.x86_64 (bug present) Config file: standard config, KVM specific config here: https://pastebin.com/skXYQeJn Qemu version: 9.1.0 We see the following turns of events with kernel-irqchip: a) A level triggered interrupt on pin 19 arrives after MMIO write from L2(CGVM) (to usb device) (not shown in above logs). b) Using kvm_nested_vmexit_inject the L2 exit reason (external interrupt) is made visible to L1. c) We see an L1 enter, which immediately invokes L2 with VMRESUME. d) L2 services the interrupt, performs MMIO write, which de-asserts the level triggered interrupt. e) Then L2 EOIs the interrupt after de-assertion -> L1 EOIs the interrupt. While for split-irqchip we see the following turn of events: a) and b) is the same. c) L1 enters and EOIs immediately, without letting L2 service the interrupt first. (Line never de-asserts) d) The interrupt gets re-inserted repeatedly -> boot crawls. [KVM Tracepoints at the bottom ] We probed the interrupt delivery path in both cases, the interrupt delivery path for both the cases is only slightly different, in the case of kernel-irqchip, `kvm_set_irq` calls `kvm_ioapic_set_irq`, while in the case of split-irqchip, `kvm_set_irq` calls `kvm_msi_set_irq`, both paths merge on `kvm_irq_delivery_to_apic`, we noticed all the args are same, we also probed path taken from there in both the cases, which is kvm_irq_delivery_to_apic → kvm_irq_delivery_to_apic_fast → kvm_apic_set_irq → __apic_accept_irq → vmx_deliver_interrupt → vmx_deliver_posted_interrupt → kvm_vcpu_trigger_posted_interrupt On split-irqchip, subsequent interrupts take different route in kvm_vcpu_trigger_posted_interrupt as interrupts are received while L1 (Windows) is running on the vcpu. We also took VMCS dumps just before kvm_enter and just after kvm_exit after the first level triggered interrupt is received, VMCS01 and VMCS12 are same just before the L1 VMENTER after the first interrupt is delivered. We observe that for both VMCS01 and VMCS12, InterruptStatus, SVI|RVI, entry/exit information, pin-based/cpu-based ctrl state are the same, yet in case of kernel-irqchip, L1 passes the interrupt to L2 with VMRESUME, and in case of split-irqchip performs EOI. [VMCS dumps at the bottom] We also checked the last MSR reads and writes on different addresses before the first level triggered interrupt, the dump is available here: https://pastebin.com/PckkM8pt, Only values we see different between kernel-irqchip and split-irqchip seems to be run to run variation. We also checked CPUID info exposed to the guest. There was only one bit change, specifically on func 40000001. A@1451335: kvm:kvm_cpuid: func 40000001 idx 0 rax 10040fb rbx 0 rcx 0 rdx 0, cpuid entry found B@1476293: kvm:kvm_cpuid: func 40000001 idx 0 rax 100c0fb rbx 0 rcx 0 rdx 0, cpuid entry found This seems to be KVM_FEATURE_MSI_EXT_DEST_ID bit, disabling KVM_FEATURE_MSI_EXT_DEST_ID does not change the behavior. [3.] Keywords: kvm,vmx,nested-virtualization,split-irqchip,windows-guest Any pointers on this will be extremely helpful thanks! Thanks, Khushit =========================================================== KVM tracepoints in the case of kernel-irqchip: CPU 0/KVM 699876 [020] 18136.763622: kvm:kvm_set_irq: gsi 19 level 1 source 0 CPU 0/KVM 699876 [020] 18136.763624: kvm:kvm_apic_accept_irq: apicid 0 vec 129 (Fixed|level) CPU 0/KVM 699876 [020] 18136.763625: kvm:kvm_apicv_accept_irq: apicid 0 vec 129 (Fixed|level) CPU 0/KVM 699876 [020] 18136.763626: kvm:kvm_ioapic_set_irq: pin 19 dst 0 vec 129 (Fixed|physical|level) CPU 0/KVM 699876 [020] 18136.763629: kvm:kvm_set_irq: gsi 11 level 0 source 0 CPU 0/KVM 699876 [020] 18136.763631: kvm:kvm_pic_set_irq: chip 1 pin 3 (edge|masked) CPU 0/KVM 699876 [020] 18136.763632: kvm:kvm_ioapic_set_irq: pin 11 dst 0 vec 255 (Fixed|physical|edge|masked) CPU 0/KVM 699876 [020] 18136.763633: kvm:kvm_set_irq: gsi 11 level 0 source 0 CPU 0/KVM 699876 [020] 18136.763634: kvm:kvm_pic_set_irq: chip 1 pin 3 (edge|masked) CPU 0/KVM 699876 [020] 18136.763634: kvm:kvm_ioapic_set_irq: pin 11 dst 0 vec 255 (Fixed|physical|edge|masked) CPU 0/KVM 699876 [020] 18136.763640: kvm:kvm_fpu: load CPU 0/KVM 699876 [020] 18136.763644: kvm:kvm_nested_vmexit_inject: reason EXTERNAL_INTERRUPT info1 0 info2 0 int_info 80000081 int_info_err 0 CPU 0/KVM 699876 [020] 18136.763647: kvm:kvm_entry: vcpu 0, rip 0xfffff8489b64142c CPU 0/KVM 699876 [020] 18136.763651: kvm:kvm_exit: reason VMRESUME rip 0xfffff8489b641308 info 0 0 CPU 0/KVM 699876 [020] 18136.763652: kvm:kvm_nested_vmenter: rip: 0xfffff8489b641308 vmcs: 0xffffffffffffffff nested_rip: 0xfffff8031964b2b6 int_ctl: 0x00000000 event_inj: 0x80000081 nested_ept=y nested_eptp: 0x00000001019a701e ————————————————————————————— L2 services the interrupt ——————————————————— CPU 0/KVM 699876 [020] 18136.763695: kvm:kvm_mmio: mmio write len 4 gpa 0x81111024 val 0x4 CPU 0/KVM 699876 [020] 18136.763696: kvm:kvm_fpu: unload CPU 0/KVM 699876 [020] 18136.763696: kvm:kvm_userspace_exit: reason KVM_EXIT_MMIO (6) CPU 0/KVM 699876 [020] 18136.763699: kvm:kvm_set_irq: gsi 19 level 0 source 0 CPU 0/KVM 699876 [020] 18136.763700: kvm:kvm_ioapic_set_irq: pin 19 dst 0 vec 129 (Fixed|physical|level) CPU 0/KVM 699876 [020] 18136.763701: kvm:kvm_set_irq: gsi 11 level 0 source 0 CPU 0/KVM 699876 [020] 18136.763701: kvm:kvm_pic_set_irq: chip 1 pin 3 (edge|masked) CPU 0/KVM 699876 [020] 18136.763702: kvm:kvm_ioapic_set_irq: pin 11 dst 0 vec 255 (Fixed|physical|edge|masked) CPU 0/KVM 699876 [020] 18136.763703: kvm:kvm_set_irq: gsi 11 level 0 source 0 CPU 0/KVM 699876 [020] 18136.763703: kvm:kvm_pic_set_irq: chip 1 pin 3 (edge|masked) CPU 0/KVM 699876 [020] 18136.763703: kvm:kvm_ioapic_set_irq: pin 11 dst 0 vec 255 (Fixed|physical|edge|masked) CPU 0/KVM 699876 [020] 18136.763705: kvm:kvm_fpu: load CPU 0/KVM 699876 [020] 18136.763705: kvm:kvm_entry: vcpu 0, rip 0xfffff8031964a708 CPU 0/KVM 699876 [020] 18136.763707: kvm:kvm_exit: reason EPT_MISCONFIG rip 0xfffff8031964a727 info 0 0 CPU 0/KVM 699876 [020] 18136.763707: kvm:kvm_nested_vmexit: CAN'T FIND FIELD "rip"<CANT FIND FIELD exit_code>vcpu 0 reason EPT_MISCONFIG rip 0xfffff8031964a727 info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000 CPU 0/KVM 699876 [020] 18136.763709: kvm:kvm_emulate_insn: 0:fffff8031964a727: 8b 56 0c CPU 0/KVM 699876 [020] 18136.763710: kvm:vcpu_match_mmio: gva 0xffffbd0c86b8a02c gpa 0x8111102c Read GPA CPU 0/KVM 699876 [020] 18136.763710: kvm:kvm_mmio: mmio unsatisfied-read len 4 gpa 0x8111102c val 0x0 CPU 0/KVM 699876 [020] 18136.763710: kvm:kvm_fpu: unload CPU 0/KVM 699876 [020] 18136.763711: kvm:kvm_userspace_exit: reason KVM_EXIT_MMIO (6) CPU 0/KVM 699876 [020] 18136.763712: kvm:kvm_fpu: load CPU 0/KVM 699876 [020] 18136.763713: kvm:kvm_mmio: mmio read len 4 gpa 0x8111102c val 0xec0 CPU 0/KVM 699876 [020] 18136.763714: kvm:kvm_entry: vcpu 0, rip 0xfffff8031964a72a CPU 0/KVM 699876 [020] 18136.763716: kvm:kvm_exit: reason MSR_WRITE rip 0xfffff8030f62d36c info 0 0 CPU 0/KVM 699876 [020] 18136.763717: kvm:kvm_nested_vmexit: CAN'T FIND FIELD "rip"<CANT FIND FIELD exit_code>vcpu 0 reason MSR_WRITE rip 0xfffff8030f62d36c info1 0x0000000000000000 info2 0x0000000000000000 intr_info 0x00000000 error_code 0x00000000 CPU 0/KVM 699876 [020] 18136.763719: kvm:kvm_nested_vmexit_inject: reason MSR_WRITE info1 0 info2 0 int_info 0 int_info_err 0 CPU 0/KVM 699876 [020] 18136.763721: kvm:kvm_entry: vcpu 0, rip 0xfffff8489b64142c CPU 0/KVM 699876 [020] 18136.763724: kvm:kvm_exit: reason MSR_WRITE rip 0xfffff8489b732daf info 0 0 CPU 0/KVM 699876 [020] 18136.763725: kvm:kvm_apic: apic_write APIC_EOI = 0x0 CPU 0/KVM 699876 [020] 18136.763725: kvm:kvm_eoi: apicid 0 vector 129 KVM tracepoints in case of split-irqchip: CPU 0/KVM 655470 [011] 17156.779254: kvm:kvm_set_irq: gsi 19 level 1 source 0 CPU 0/KVM 655470 [011] 17156.779255: kvm:kvm_msi_set_irq: dst 0 vec 129 (Fixed|physical|level) CPU 0/KVM 655470 [011] 17156.779256: kvm:kvm_apic_accept_irq: apicid 0 vec 129 (Fixed|level) CPU 0/KVM 655470 [011] 17156.779256: kvm:kvm_apicv_accept_irq: apicid 0 vec 129 (Fixed|level) CPU 0/KVM 655470 [011] 17156.779263: kvm:kvm_fpu: load CPU 0/KVM 655470 [011] 17156.779266: kvm:kvm_nested_vmexit_inject: reason EXTERNAL_INTERRUPT info1 0 info2 0 int_info 80000081 int_info_err 0 CPU 0/KVM 655470 [011] 17156.779269: kvm:kvm_entry: vcpu 0, rip 0xfffff80653e4142c CPU 0/KVM 655470 [011] 17156.779271: kvm:kvm_exit: reason MSR_WRITE rip 0xfffff80653f32daf info 0 0 CPU 0/KVM 655470 [011] 17156.779272: kvm:kvm_apic: apic_write APIC_EOI = 0x0 CPU 0/KVM 655470 [011] 17156.779272: kvm:kvm_eoi: apicid 0 vector 129 CPU 0/KVM 655470 [011] 17156.779273: kvm:kvm_msr: msr_write 40000070 = 0x0 CPU 0/KVM 655470 [011] 17156.779274: kvm:kvm_fpu: unload CPU 0/KVM 655470 [011] 17156.779274: kvm:kvm_userspace_exit: reason KVM_EXIT_IOAPIC_EOI (26) VMCS dump for kernel-irqchip: CPU 0/KVM-251862 [019] 6384.132433: bprint: vmx_dump_01_02_12: KVM/VMX: ===== autodump (VM-ENTER) vcpu=0 ===== CPU 0/KVM-251862 [019] 6384.132433: bputs: vmx_dump_01_02_12: KVM/VMX: [CURRENT] CPU 0/KVM-251862 [019] 6384.132433: bputs: vmx_dump_01_02_12: KVM/VMC: [CURRENT] is vmc01 CPU 0/KVM-251862 [019] 6384.132433: bprint: dump_vmcs: VMCS 0xffff906388d0c000, last attempted VM-entry on CPU 19 CPU 0/KVM-251862 [019] 6384.132433: bputs: dump_vmcs: *** Guest State *** CPU 0/KVM-251862 [019] 6384.132433: bprint: dump_vmcs: CR0: actual=0x0000000080010031, shadow=0x0000000080010031, gh_mask=fffffffffffefff7 CPU 0/KVM-251862 [019] 6384.132434: bprint: dump_vmcs: CR4: actual=0x00000000000422e0, shadow=0x00000000000422e0, gh_mask=fffffffffffef871 CPU 0/KVM-251862 [019] 6384.132434: bprint: dump_vmcs: CR3 = 0x0000000101945000 CPU 0/KVM-251862 [019] 6384.132434: bprint: dump_vmcs: PDPTR0 = 0x0000000065632001 PDPTR1 = 0x0000000065631001 CPU 0/KVM-251862 [019] 6384.132434: bprint: dump_vmcs: PDPTR2 = 0x0000000065630001 PDPTR3 = 0x000000006562f001 CPU 0/KVM-251862 [019] 6384.132435: bprint: dump_vmcs: RSP = 0xffffe70000005fc0 RIP = 0xfffff8390c241308 CPU 0/KVM-251862 [019] 6384.132435: bprint: dump_vmcs: RFLAGS=0x00000002 DR7 = 0x0000000000000400 CPU 0/KVM-251862 [019] 6384.132435: bprint: vmx_dump_dtsel: GDTR: limit=0x0000ffff, base=0xfffff8390c0232b0 CPU 0/KVM-251862 [019] 6384.132435: bprint: vmx_dump_dtsel: IDTR: limit=0x0000ffff, base=0xfffff8390c0220b0 CPU 0/KVM-251862 [019] 6384.132435: bprint: vmx_dump_sel.constprop.0: TR: sel=0x0030, attr=0x0008b, limit=0x00000067, base=0xfffff8390c123320 CPU 0/KVM-251862 [019] 6384.132436: bprint: dump_vmcs: DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 CPU 0/KVM-251862 [019] 6384.132436: bprint: dump_vmcs: Interruptibility = 00000000 ActivityState = 00000000 CPU 0/KVM-251862 [019] 6384.132436: bprint: dump_vmcs: InterruptStatus = 5100 CPU 0/KVM-251862 [019] 6384.132436: bprint: vmx_dump_msrs: MSR guest autoload: CPU 0/KVM-251862 [019] 6384.132436: bprint: vmx_dump_msrs: 0: msr=0x00000600 value=0x0000000000000000 CPU 0/KVM-251862 [019] 6384.132437: bputs: dump_vmcs: *** Host State *** CPU 0/KVM-251862 [019] 6384.132437: bprint: dump_vmcs: RIP = 0xffffffffc0d89630 RSP = 0xffffb5c59424bce0 CPU 0/KVM-251862 [019] 6384.132437: bprint: dump_vmcs: GDTBase=fffffe3f5127d000 IDTBase=fffffe0000000000 CPU 0/KVM-251862 [019] 6384.132437: bprint: dump_vmcs: CR0=0000000080050033 CR3=0000000183c24001 CR4=0000000000772ef0 CPU 0/KVM-251862 [019] 6384.132437: bprint: dump_vmcs: PAT = 0x0407050600070106 CPU 0/KVM-251862 [019] 6384.132437: bprint: vmx_dump_msrs: MSR host autoload: CPU 0/KVM-251862 [019] 6384.132438: bprint: vmx_dump_msrs: 0: msr=0x00000600 value=0xfffffe3f51297000 CPU 0/KVM-251862 [019] 6384.132438: bputs: dump_vmcs: *** Control State *** CPU 0/KVM-251862 [019] 6384.132438: bprint: dump_vmcs: CPUBased=0xb5a26dfa SecondaryExec=0x821217eb TertiaryExec=0x0000000000000010 CPU 0/KVM-251862 [019] 6384.132438: bprint: dump_vmcs: PinBased=0x000000ff EntryControls=000053ff ExitControls=000befff CPU 0/KVM-251862 [019] 6384.132438: bprint: dump_vmcs: ExceptionBitmap=000600c2 PFECmask=00000000 PFECmatch=00000000 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: reason=00000018 qualification=0000000000000000 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: IDTVectoring: info=00000000 errcode=00000000 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: TSC Offset = 0xfffff447b4a645a1 CPU 0/KVM-251862 [019] 6384.132439: bprint: dump_vmcs: TSC Multiplier = 0x0001000000000000 CPU 0/KVM-251862 [019] 6384.132440: bprint: dump_vmcs: SVI|RVI = 51|00 CPU 0/KVM-251862 [019] 6384.132440: bprint: dump_vmcs: TPR Threshold = 0x00 CPU 0/KVM-251862 [019] 6384.132440: bprint: dump_vmcs: APIC-access addr = 0x000000024ea83000 CPU 0/KVM-251862 [019] 6384.132440: bprint: dump_vmcs: virt-APIC addr = 0x00000046212ed000 CPU 0/KVM-251862 [019] 6384.132440: bprint: dump_vmcs: PostedIntrVec = 0xf2 CPU 0/KVM-251862 [019] 6384.132441: bprint: dump_vmcs: EPT pointer = 0x00000040a0db305e CPU 0/KVM-251862 [019] 6384.132441: bprint: dump_vmcs: PLE Gap=00000080 Window=00001000 CPU 0/KVM-251862 [019] 6384.132441: bprint: dump_vmcs: Virtual processor ID = 0x0011 CPU 0/KVM-251862 [019] 6384.132441: bputs: vmx_dump_01_02_12: KVM/VMX: [VMCS12] CPU 0/KVM-251862 [019] 6384.132441: bprint: vmx_dump_01_02_12: ENTRY_INTR_INFO=0xd1 ENTRY_LEN=0 EXIT_REASON=0x1 EXIT_INTR_INFO=0x80000051 EXIT_INTR_ERR=0 CPU 0/KVM-251862 [019] 6384.132442: bprint: vmx_dump_01_02_12: IDT_VEC_INFO=0 IDT_VEC_ERR=0 EXIT_LEN=0x3 VMX_INSN_INFO=0 CPU 0/KVM-251862 [019] 6384.132442: bprint: vmx_dump_01_02_12: GUEST_INTR_STATUS=0 TPR_THRESH=0 VAPIC=0x116688000 PIDESC=0 PINV=0 CPU 0/KVM-251862 [019] 6384.132442: bprint: vmx_dump_01_02_12: EOI_EXIT_BITMAPS: [0]=0 [1]=0 [2]=0 [3]=0 CPU 0/KVM-251862 [019] 6384.132442: bprint: vmx_dump_01_02_12: PINBASED=0x3f CPU_BASED=0xb6206dfa CPU_BASED2=0x1010ae VM_ENTRY_CTRL=0x13ff VM_EXIT_CTRL=0x3efff CPU 0/KVM-251862 [019] 6384.132443: bprint: vmx_dump_01_02_12: RIP=0xfffff8012b5f50c2 RSP=0xfffffc8791b29fd0 RFLAGS=0x40213 INTR_STATE=0 ACTV=0 CPU 0/KVM-251862 [019] 6384.132443: bprint: vmx_dump_01_02_12: CR0=0x80050033 CR3=0x119167000 CR4=0x352ef8 EPTP=0x1019ab01e CPU 0/KVM-251862 [019] 6384.132443: bputs: vmx_dump_01_02_12: KVM/VMX: ===================================== VMCS dump for split-irqchip, CPU 0/KVM-453395 [001] 12461.840701: bprint: vmx_dump_01_02_12: KVM/VMX: ===== autodump (VM-ENTER) vcpu=0 ===== CPU 0/KVM-453395 [001] 12461.840702: bputs: vmx_dump_01_02_12: KVM/VMX: [CURRENT] CPU 0/KVM-453395 [001] 12461.840702: bputs: vmx_dump_01_02_12: KVM/VMC: [CURRENT] is vmc01 CPU 0/KVM-453395 [001] 12461.840702: bprint: dump_vmcs: VMCS 0xffff906904c87000, last attempted VM-entry on CPU 1 CPU 0/KVM-453395 [001] 12461.840702: bputs: dump_vmcs: *** Guest State *** CPU 0/KVM-453395 [001] 12461.840702: bprint: dump_vmcs: CR0: actual=0x0000000080010031, shadow=0x0000000080010031, gh_mask=fffffffffffefff7 CPU 0/KVM-453395 [001] 12461.840702: bprint: dump_vmcs: CR4: actual=0x00000000000422e0, shadow=0x00000000000422e0, gh_mask=fffffffffffef871 CPU 0/KVM-453395 [001] 12461.840703: bprint: dump_vmcs: CR3 = 0x0000000101944000 CPU 0/KVM-453395 [001] 12461.840703: bprint: dump_vmcs: PDPTR0 = 0x0000000065632001 PDPTR1 = 0x0000000065631001 CPU 0/KVM-453395 [001] 12461.840703: bprint: dump_vmcs: PDPTR2 = 0x0000000065630001 PDPTR3 = 0x000000006562f001 CPU 0/KVM-453395 [001] 12461.840703: bprint: dump_vmcs: RSP = 0xffffe70000005fc0 RIP = 0xfffff808b8441308 CPU 0/KVM-453395 [001] 12461.840703: bprint: dump_vmcs: RFLAGS=0x00000002 DR7 = 0x0000000000000400 CPU 0/KVM-453395 [001] 12461.840704: bprint: vmx_dump_dtsel: GDTR: limit=0x0000ffff, base=0xfffff808b82232b0 CPU 0/KVM-453395 [001] 12461.840704: bprint: vmx_dump_dtsel: IDTR: limit=0x0000ffff, base=0xfffff808b82220b0 CPU 0/KVM-453395 [001] 12461.840704: bprint: vmx_dump_sel.constprop.0: TR: sel=0x0030, attr=0x0008b, limit=0x00000067, base=0xfffff808b8323320 CPU 0/KVM-453395 [001] 12461.840704: bprint: dump_vmcs: DebugCtl = 0x0000000000000000 DebugExceptions = 0x0000000000000000 CPU 0/KVM-453395 [001] 12461.840704: bprint: dump_vmcs: Interruptibility = 00000000 ActivityState = 00000000 CPU 0/KVM-453395 [001] 12461.840705: bprint: dump_vmcs: InterruptStatus = 5100 CPU 0/KVM-453395 [001] 12461.840705: bprint: vmx_dump_msrs: MSR guest autoload: CPU 0/KVM-453395 [001] 12461.840705: bprint: vmx_dump_msrs: 0: msr=0x00000600 value=0x0000000000000000 CPU 0/KVM-453395 [001] 12461.840705: bputs: dump_vmcs: *** Host State *** CPU 0/KVM-453395 [001] 12461.840705: bprint: dump_vmcs: RIP = 0xffffffffc0d89630 RSP = 0xffffb5c5a0117ce0 CPU 0/KVM-453395 [001] 12461.840705: bprint: dump_vmcs: GDTBase=fffffe5a3972e000 IDTBase=fffffe0000000000 CPU 0/KVM-453395 [001] 12461.840705: bprint: dump_vmcs: CR0=0000000080050033 CR3=0000000109cee003 CR4=0000000000772ef0 CPU 0/KVM-453395 [001] 12461.840706: bprint: dump_vmcs: PAT = 0x0407050600070106 CPU 0/KVM-453395 [001] 12461.840706: bprint: vmx_dump_msrs: MSR host autoload: CPU 0/KVM-453395 [001] 12461.840706: bprint: vmx_dump_msrs: 0: msr=0x00000600 value=0xfffffe5a39748000 CPU 0/KVM-453395 [001] 12461.840706: bputs: dump_vmcs: *** Control State *** CPU 0/KVM-453395 [001] 12461.840706: bprint: dump_vmcs: CPUBased=0xb5a26dfa SecondaryExec=0x821217eb TertiaryExec=0x0000000000000010 CPU 0/KVM-453395 [001] 12461.840706: bprint: dump_vmcs: PinBased=0x000000ff EntryControls=000053ff ExitControls=000befff CPU 0/KVM-453395 [001] 12461.840707: bprint: dump_vmcs: ExceptionBitmap=000600c2 PFECmask=00000000 PFECmatch=00000000 CPU 0/KVM-453395 [001] 12461.840707: bprint: dump_vmcs: VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000 CPU 0/KVM-453395 [001] 12461.840707: bprint: dump_vmcs: VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 CPU 0/KVM-453395 [001] 12461.840707: bprint: dump_vmcs: reason=00000018 qualification=0000000000000000 CPU 0/KVM-453395 [001] 12461.840708: bprint: dump_vmcs: IDTVectoring: info=00000000 errcode=00000000 CPU 0/KVM-453395 [001] 12461.840708: bprint: dump_vmcs: TSC Offset = 0xffffe9399281aa53 CPU 0/KVM-453395 [001] 12461.840708: bprint: dump_vmcs: TSC Multiplier = 0x0001000000000000 CPU 0/KVM-453395 [001] 12461.840708: bprint: dump_vmcs: SVI|RVI = 51|00 CPU 0/KVM-453395 [001] 12461.840708: bprint: dump_vmcs: TPR Threshold = 0x00 CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: APIC-access addr = 0x000000012e1ad000 CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: virt-APIC addr = 0x0000004605554000 CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: PostedIntrVec = 0xf2 CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: EPT pointer = 0x00000001bdfa805e CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: PLE Gap=00000080 Window=00001000 CPU 0/KVM-453395 [001] 12461.840709: bprint: dump_vmcs: Virtual processor ID = 0x0011 CPU 0/KVM-453395 [001] 12461.840710: bputs: vmx_dump_01_02_12: KVM/VMX: [VMCS12] CPU 0/KVM-453395 [001] 12461.840710: bprint: vmx_dump_01_02_12: ENTRY_INTR_INFO=0xd1 ENTRY_LEN=0 EXIT_REASON=0x1 EXIT_INTR_INFO=0x80000051 EXIT_INTR_ERR=0 CPU 0/KVM-453395 [001] 12461.840710: bprint: vmx_dump_01_02_12: IDT_VEC_INFO=0 IDT_VEC_ERR=0 EXIT_LEN=0x5 VMX_INSN_INFO=0 CPU 0/KVM-453395 [001] 12461.840710: bprint: vmx_dump_01_02_12: GUEST_INTR_STATUS=0 TPR_THRESH=0 VAPIC=0x116687000 PIDESC=0 PINV=0 CPU 0/KVM-453395 [001] 12461.840711: bprint: vmx_dump_01_02_12: EOI_EXIT_BITMAPS: [0]=0 [1]=0 [2]=0 [3]=0 CPU 0/KVM-453395 [001] 12461.840711: bprint: vmx_dump_01_02_12: PINBASED=0x3f CPU_BASED=0xb6206dfa CPU_BASED2=0x1010ae VM_ENTRY_CTRL=0x13ff VM_EXIT_CTRL=0x3efff CPU 0/KVM-453395 [001] 12461.840711: bprint: vmx_dump_01_02_12: RIP=0xfffff8035aff73fc RSP=0xffffc18d6b729f58 RFLAGS=0x40246 INTR_STATE=0 ACTV=0 CPU 0/KVM-453395 [001] 12461.840711: bprint: vmx_dump_01_02_12: CR0=0x80050033 CR3=0x11912e000 CR4=0x352ef8 EPTP=0x1019aa01e CPU 0/KVM-453395 [001] 12461.840711: bputs: vmx_dump_01_02_12: KVM/VMX: ===================================== The full VMCS dumps + kvm tracepoints near the first level triggered interrupt for kernel-irqchip are here: https://pastebin.com/2tnN6Xfy, and for split-irqchip are here: https://pastebin.com/K1BKZYvv
