On Thu, Jul 24, 2025 at 02:52:08PM +0200, Stefano Garzarella wrote:
> On Thu, 24 Jul 2025 at 14:48, Breno Leitao <leitao@xxxxxxxxxx> wrote:
> >
> > On Thu, Jul 24, 2025 at 09:44:38AM +0100, Will Deacon wrote:
> > > > > On Thu, 24 Jul 2025 at 09:48, Michael S. Tsirkin <mst@xxxxxxxxxx> wrote:
> > > > > >
> > > > > > On Wed, Jul 23, 2025 at 08:04:42AM -0700, Breno Leitao wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I've seen a crash in linux-next for a while on my arm64 server, and
> > > > > > > I decided to report.
> > > > > > >
> > > > > > > While running stress-ng on linux-next, I see the crash below.
> > > > > > >
> > > > > > > This is happening in a kernel configure with some debug options (KASAN,
> > > > > > > LOCKDEP and KMEMLEAK).
> > > > > > >
> > > > > > > Basically running stress-ng in a loop would crash the host in 15-20
> > > > > > > minutes:
> > > > > > > # while (true); do stress-ng -r 10 -t 10; done
> > > > > > >
> > > > > > > >From the early warning "virt_to_phys used for non-linear address",
> > > > >
> > > > > mmm, we recently added nonlinear SKBs support in vhost-vsock [1],
> > > > > @Will can this issue be related?
> > > >
> > > > Good point.
> > > >
> > > > Breno, if bisecting is too much trouble, would you mind testing the commits
> > > > c76f3c4364fe523cd2782269eab92529c86217aa
> > > > and
> > > > c7991b44d7b44f9270dec63acd0b2965d29aab43
> > > > and telling us if this reproduces?
> > >
> > > That's definitely worth doing, but we should be careful not to confuse
> > > the "non-linear address" from the warning (which refers to virtual
> > > addresses that lie outside of the linear mapping of memory, e.g. in the
> > > vmalloc space) and "non-linear SKBs" which refer to SKBs with fragment
> > > pages.
> >
> > I've tested both commits above, and I see the crash on both commits
> > above, thus, the problem reproduces in both cases. The only difference
> > I noted is the fact that I haven't seen the warning before the crash.
> >
> >
> > Log against c76f3c4364fe ("vhost/vsock: Avoid allocating
> > arbitrarily-sized SKBs")
> >
> > Unable to handle kernel paging request at virtual address 0000001fc0000048
> > Mem abort info:
> > ESR = 0x0000000096000005
> > EC = 0x25: DABT (current EL), IL = 32 bits
> > SET = 0, FnV = 0
> > EA = 0, S1PTW = 0
> > FSC = 0x05: level 1 translation fault
> > Data abort info:
> > ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
> > CM = 0, WnR = 0, TnD = 0, TagAccess = 0
> > GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
> > user pgtable: 64k pages, 48-bit VAs, pgdp=0000000cdcf2da00
> > [0000001fc0000048] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
> > Internal error: Oops: 0000000096000005 [#1] SMP
> > Modules linked in: vfio_iommu_type1 vfio md4 crc32_cryptoapi ghash_generic unix_diag vhost_net tun vhost vhost_iotlb tap mpls_gso mpls_iptunnel mpls_router fou sch_fq ghes_edac tls tcp_diag inet_diag act_gact cls_bpf nvidia_c
> > CPU: 34 UID: 0 PID: 1727297 Comm: stress-ng-dev Kdump: loaded Not tainted 6.16.0-rc6-upstream-00027-gc76f3c4364fe #19 NONE
> > pstate: 23401009 (nzCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
> > pc : kfree+0x48/0x2a8
> > lr : vhost_dev_cleanup+0x138/0x2b8 [vhost]
> > sp : ffff80013a0cfcd0
> > x29: ffff80013a0cfcd0 x28: ffff0008fd0b6240 x27: 0000000000000000
> > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000
> > x23: 00000000040e001f x22: ffffffffffffffff x21: ffff00014f1d4ac0
> > x20: 0000000000000001 x19: ffff00014f1d0000 x18: 0000000000000000
> > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> > x14: 000000000000001f x13: 000000000000000f x12: 0000000000000001
> > x11: 0000000000000000 x10: 0000000000000402 x9 : ffffffdfc0000000
> > x8 : 0000001fc0000040 x7 : 0000000000000000 x6 : 0000000000000000
> > x5 : ffff000141931840 x4 : 0000000000000000 x3 : 0000000000000008
> > x2 : ffffffffffffffff x1 : ffffffffffffffff x0 : 0000000000010000
> > Call trace:
> > kfree+0x48/0x2a8 (P)
> > vhost_dev_cleanup+0x138/0x2b8 [vhost]
> > vhost_net_release+0xa0/0x1a8 [vhost_net]
>
> But here is the vhost_net, so I'm confused now.
> Do you see the same (vhost_net) also on 9798752 ("Add linux-next
> specific files for 20250721") ?
I will need to reproduce, but, looking at my logs, I see the following
against: c76f3c4364fe ("vhost/vsock: Avoid allocating arbitrarily-sized SKBs").
The logs are a bit intermixed, probably there were multiple CPUs hitting
the same code path.
virt_to_phys used for non-linear address: 000000001b662678 (0xffe61984a460)
WARNING: CPU: 15 PID: 112846 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8
Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E)
CPU: 15 UID: 0 PID: 112846 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none)
Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
pc : __virt_to_phys+0x80/0xa8
lr : __virt_to_phys+0x7c/0xa8
sp : ffff8001184d7a30
x29: ffff8001184d7a30 x28: 00000000000045d8 x27: 1fffe000e7e88014
x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013
x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098
x20: ffffff1000080000 x19: 0000ffe61984a460 x18: 0000000000000002
x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001
x14: 1fffe006d52e90f2 x13: 0000000000000000 x12: 0000000000000000
x11: ffff6006d52e90f3 x10: 0000000000000002 x9 : cfc659a21c727d00
x8 : ffff800083c19000 x7 : 0000000000000001 x6 : 0000000000000001
x5 : ffff8001184d7398 x4 : ffff800084866d60 x3 : ffff8000805fdd94
x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b
Call trace:
__virt_to_phys+0x80/0xa8 (P)
kfree+0xac/0x4b0
vhost_dev_cleanup+0x484/0x8b0 [vhost]
vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock]
__fput+0x2b4/0x608
fput_close_sync+0xe8/0x1e0
__arm64_sys_close+0x84/0xd0
invoke_syscall+0x8c/0x208
do_el0_svc+0x128/0x1a0
el0_svc+0x58/0x160
el0t_64_sync_handler+0x78/0x108
el0t_64_sync+0x198/0x1a0
irq event stamp: 0
hardirqs last enabled at (0): [<0000000000000000>] 0x0
hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8
softirqs last enabled at (0): [<ffff8000801d879c>] copy_process+0xd8c/0x29f8
softirqs last disabled at (0): [<0000000000000000>] 0x0
---[ end trace 0000000000000000 ]---
Unable to handle kernel paging request at virtual address 0000040053791288
------------[ cut here ]------------
lr : kfree+0xac/0x4b0
virt_to_phys used for non-linear address: 00000000290839fd (0x2500000000)
WARNING: CPU: 41 PID: 112845 at arch/arm64/mm/physaddr.c:15 __virt_to_phys+0x80/0xa8
Modules linked in: vhost_vsock(E) vhost(E) vhost_iotlb(E) ghes_edac(E) tls(E) act_gact(E) cls_bpf(E) tcp_diag(E) inet_diag(E) ipmi_ssif(E) ipmi_devintf(E) ipmi_msghandler(E) sch_fq_codel(E) drm(E) backlight(E) drm_panel_orientation_quirks(E) acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E)
CPU: 41 UID: 0 PID: 112845 Comm: stress-ng-dev Kdump: loaded Tainted: G W E N 6.16.0-rc6-upstream-00027-gc76f3c4364fe #16 PREEMPT(none)
x23: 0000000000000001
Tainted: [W]=WARN, [E]=UNSIGNED_MODULE, [N]=TEST
pstate: 63401009 (nZCv daif +PAN -UAO +TCO +DIT +SSBS BTYPE=--)
pc : __virt_to_phys+0x80/0xa8
lr : __virt_to_phys+0x7c/0xa8
sp : ffff8001a8277a30
x29: ffff8001a8277a30 x28: 00000000000045d8 x27: 1fffe000e7e3c014
x26: 1fffe000e7e3c8f7 x25: ffff0007bcff0000 x24: 1fffe000e7e3c013
x23: 0000000000000000 x22: 0000002500000000 x21: ffff00073f1e0098
x20: ffffff1000080000 x19: 0000002500000000 x18: 0000000000000004
x17: 00000000ffffffff x16: 0000000000000001 x15: 0000000000000001
x14: 1fffe006d53920f2 x13: 0000000000000000 x12: 0000000000000000
sp : ffff8001184d7a50
x29: ffff8001184d7a60 x28: 00000000000045d8 x27: 1fffe000e7e88014
x26: 1fffe000e7e888f7 x25: ffff0007e578bf00 x24: 1fffe000e7e88013
x23: 0000000000000000 x22: 0000ffe61984a460 x21: ffff00073f440098
x20: 0000040053791280 x19: ffff80000d0b8bbc x18: 0000000000000002
x17: 6666783028203837 x16: 3632363662313030 x15: 0000000000000001
x22: 0000ffe6199a459d
x14: 1fffe006d52e90f2
x21: ffff00073f7f0098
x20: ffffff1000080000 x19: 0000ffe6199a459d x18: 0000000000000004
x17: 54455320203b2d2c x16: 0000000000000011 x15: 0000000000000001
x14: 1fffe006d52bb8f2 x13: 0000000000000000 x12: 0000000000000000
x11: ffff6006d52bb8f3 x10: dfff800000000000 x9 : 77521a2bd3e0be00
x8 : ffff800083c19000 x7 : 0000000000000000 x6 : ffff80008036bc2c
x5 : 0000000000000000 x4 : 0000000000000001 x3 : ffff8000805fdd94
x2 : 0000000000000001 x1 : 0000000000000004 x0 : 000000000000004b
Call trace:
__virt_to_phys+0x80/0xa8 (P)
kfree+0xac/0x4b0
vhost_dev_cleanup+0x484/0x8b0 [vhost]
vhost_vsock_dev_release+0x2f4/0x358 [vhost_vsock]
__fput+0x2b4/0x608
x11: ffff6006d53920f3
fput_close_sync+0xe8/0x1e0
__arm64_sys_close+0x84/0xd0
invoke_syscall+0x8c/0x208
do_el0_svc+0x128/0x1a0
el0_svc+0x58/0x160
el0t_64_sync_handler+0x78/0x108
el0t_64_sync+0x198/0x1a0
irq event stamp: 0
x13: 0000000000000000
hardirqs last enabled at (0): [<0000000000000000>] 0x0
hardirqs last disabled at (0): [<ffff8000801d876c>] copy_process+0xd5c/0x29f8
> The initial report contained only vhost_vsock traces IIUC, so I'm
> suspecting something in the vhost core.
Right, it seems we are hitting the same code path, on on both
vhost_vsock and vhost_net.
