On Mon, Jul 14, 2025 at 10:31:43AM -0400, Mike Snitzer wrote:
> On Mon, Jul 14, 2025 at 01:56:13PM +1000, NeilBrown wrote:
> >
> > This wait_var_event_spinlock() in nfs_uuid_put() is waiting for the
> > wakeup signalled at the end of nfs_close_local_fh(). That
> > wake_up_var_locked() uses &nfl->nfs_uuid, so the waiter must use the
> > same address, else nfs_uuid_put() could wait indefinitely causing
> > various problems.
> >
> > Fixes: 21fb44034695 ("nfs_localio: protect race between nfs_uuid_put() and nfs_close_local_fh()")
> > Reported-by: Mike Snitzer <snitzer@xxxxxxxxxx>
> > Signed-off-by: NeilBrown <neil@xxxxxxxxxx>
> > ---
> > fs/nfs_common/nfslocalio.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/fs/nfs_common/nfslocalio.c b/fs/nfs_common/nfslocalio.c
> > index 05c7c16e37ab..bc8dcfb256a3 100644
> > --- a/fs/nfs_common/nfslocalio.c
> > +++ b/fs/nfs_common/nfslocalio.c
> > @@ -177,7 +177,7 @@ static bool nfs_uuid_put(nfs_uuid_t *nfs_uuid)
> > /* nfs_close_local_fh() is doing the
> > * close and we must wait. until it unlinks
> > */
> > - wait_var_event_spinlock(nfl,
> > + wait_var_event_spinlock(&nfl->nfs_uuid,
> > list_first_entry_or_null(
> > &nfs_uuid->files,
> > struct nfs_file_localio,
> > --
> > 2.49.0
> >
> >
>
> Makes sense:
>
> Acked-by: Mike Snitzer <snitzer@xxxxxxxxxx>
>
> And I _will_ try to get this tested at the scale I was able to test
> late last week. Will let you and others know (hopefully within the
> next 24h).
Unfortunately this fix doesn't resolve the hang I easily experience
simply by running fio like I reported here (followed by attempted NFSD
shutdown):
https://lore.kernel.org/linux-nfs/aG0pJXVtApZ9C5vy@xxxxxxxxxx/
After fio runs, 'systemctl stop proc-fs-nfsd.mount' hangs:
[ 737.997724] INFO: task rpc.nfsd:9947 blocked for more than 122 seconds.
[ 737.997984] Not tainted 6.12.24.9.hs.snitm+ #16
[ 737.998155] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 737.998304] task:rpc.nfsd state:D stack:0 pid:9947 tgid:9947 ppid:1 flags:0x00004006
[ 737.998453] Call Trace:
[ 737.998597] <TASK>
[ 737.998781] __schedule+0x26d/0x530
[ 737.998925] schedule+0x27/0xa0
[ 737.999057] schedule_timeout+0x14e/0x160
[ 737.999193] ? svc_destroy+0xce/0x160 [sunrpc]
[ 737.999378] ? lockd_put+0x5f/0x90 [lockd]
[ 737.999518] __wait_for_common+0x8f/0x1d0
[ 737.999667] ? __pfx_schedule_timeout+0x10/0x10
[ 737.999801] nfsd_destroy_serv+0x13f/0x1a0 [nfsd]
[ 737.999976] nfsd_svc+0xe0/0x170 [nfsd]
[ 738.000136] write_threads+0xc3/0x190 [nfsd]
[ 738.000293] ? simple_transaction_get+0xc2/0xe0
[ 738.000421] ? __pfx_write_threads+0x10/0x10 [nfsd]
[ 738.000578] nfsctl_transaction_write+0x47/0x80 [nfsd]
[ 738.000780] vfs_write+0xfa/0x420
[ 738.000906] ksys_write+0x63/0xe0
[ 738.001030] do_syscall_64+0x7d/0x160
[ 738.001152] ? do_user_addr_fault+0x341/0x6b0
[ 738.001278] ? exc_page_fault+0x70/0x160
[ 738.001400] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 738.001526] RIP: 0033:0x7fd836efd617
[ 738.001661] RSP: 002b:00007ffee6d1ff18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 738.001788] RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007fd836efd617
[ 738.001914] RDX: 0000000000000002 RSI: 0000558592fc1c20 RDI: 0000000000000003
[ 738.002041] RBP: 0000000000000003 R08: 0000000000000000 R09: 00007ffee6d1fdb0
[ 738.002169] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558592fc1c20
[ 738.002298] R13: 00007fd83711a6c8 R14: 00007ffee6d1ffe0 R15: 00000000ffffffff
[ 738.002429] </TASK>
So I stand by my _strong_ suggestion to revert your LOCALIO changes
in time for 6.16-rc7 (changes that were merged for 6.16), series here:
https://lore.kernel.org/linux-nfs/20250714031359.10192-1-snitzer@xxxxxxxxxx/
At this point they'd even be difficult to justify landing during the
6.17 merge window let alone 6.16 final.
Just the messenger here.. sorry.
Mike
