On Tue, Sep 02, 2025 at 01:59:12PM -0400, Chuck Lever wrote: > On 9/2/25 11:56 AM, Chuck Lever wrote: > > On 8/30/25 1:38 PM, Mike Snitzer wrote: > > >> dt (j:1 t:1): File System Information: > >> dt (j:1 t:1): Mounted from device: 192.168.0.105:/hs_test > >> dt (j:1 t:1): Mounted on directory: /mnt/hs_test > >> dt (j:1 t:1): Filesystem type: nfs4 > >> dt (j:1 t:1): Filesystem options: rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,nconnect=16,port=20491,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.106,local_lock=none,addr=192.168.0.105 > > > > I haven't been able to reproduce a similar failure in my lab with > > NFSv4.2 over RDMA with FDR InfiniBand. I've run dt 6-7 times, all > > successful. Also, for shit giggles, I tried the fsx-based subtests in > > fstests, no new failures there either. The export is xfs on an NVMe > > add-on card; server uses direct I/O for READ and page cache for WRITE. > > > > Notice the mount options for your test run: "proto=tcp" and > > "nconnect=16". Even if your network fabric is RoCE, "proto=tcp" will > > not use RDMA at all; it will use bog standard TCP/IP on your ultra > > fast Ethernet network. > > > > What should I try next? I can apply 2/2 or add "nconnect" or move the > > testing to my RoCE fabric after lunch and keep poking at it. Hmm, I'll have to check with the Hammerspace performance team to understand how RDMA used if the client mount has proto=tcp. Certainly surprising, thanks for noticing/reporting this aspect. I also cannot reproduce on a normal tcp mount and testbed. This frankenbeast of a fast "RDMA" network that is misconfigured to use proto=tcp is the only testbed where I've seen this dt data mismatch. > > Or, I could switch to TCP. Suggestions welcome. > > The client is not sending any READ procedures/operations to the server. > The following is NFSv3 for clarity, but NFSv4.x results are similar: > > nfsd-1669 [003] 1466.634816: svc_process: > addr=192.168.2.67 xid=0x7b2a6274 service=nfsd vers=3 proc=NULL > nfsd-1669 [003] 1466.635389: svc_process: > addr=192.168.2.67 xid=0x7d2a6274 service=nfsd vers=3 proc=FSINFO > nfsd-1669 [003] 1466.635420: svc_process: > addr=192.168.2.67 xid=0x7e2a6274 service=nfsd vers=3 proc=PATHCONF > nfsd-1669 [003] 1466.635451: svc_process: > addr=192.168.2.67 xid=0x7f2a6274 service=nfsd vers=3 proc=GETATTR > nfsd-1669 [003] 1466.635486: svc_process: > addr=192.168.2.67 xid=0x802a6274 service=nfsacl vers=3 proc=NULL > nfsd-1669 [003] 1466.635558: svc_process: > addr=192.168.2.67 xid=0x812a6274 service=nfsd vers=3 proc=FSINFO > nfsd-1669 [003] 1466.635585: svc_process: > addr=192.168.2.67 xid=0x822a6274 service=nfsd vers=3 proc=GETATTR > nfsd-1669 [003] 1470.029208: svc_process: > addr=192.168.2.67 xid=0x832a6274 service=nfsd vers=3 proc=ACCESS > nfsd-1669 [003] 1470.029255: svc_process: > addr=192.168.2.67 xid=0x842a6274 service=nfsd vers=3 proc=LOOKUP > nfsd-1669 [003] 1470.029296: svc_process: > addr=192.168.2.67 xid=0x852a6274 service=nfsd vers=3 proc=FSSTAT > nfsd-1669 [003] 1470.039715: svc_process: > addr=192.168.2.67 xid=0x862a6274 service=nfsacl vers=3 proc=GETACL > nfsd-1669 [003] 1470.039758: svc_process: > addr=192.168.2.67 xid=0x872a6274 service=nfsd vers=3 proc=CREATE > nfsd-1669 [003] 1470.040091: svc_process: > addr=192.168.2.67 xid=0x882a6274 service=nfsd vers=3 proc=WRITE > nfsd-1669 [003] 1470.040469: svc_process: > addr=192.168.2.67 xid=0x892a6274 service=nfsd vers=3 proc=GETATTR > nfsd-1669 [003] 1470.040503: svc_process: > addr=192.168.2.67 xid=0x8a2a6274 service=nfsd vers=3 proc=ACCESS > nfsd-1669 [003] 1470.041867: svc_process: > addr=192.168.2.67 xid=0x8b2a6274 service=nfsd vers=3 proc=FSSTAT > nfsd-1669 [003] 1470.042109: svc_process: > addr=192.168.2.67 xid=0x8c2a6274 service=nfsd vers=3 proc=REMOVE > > So I'm probably missing some setting on the reproducer/client. > > /mnt from klimt.ib.1015granger.net:/export/fast > Flags: rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard, > fatal_neterrors=none,proto=rdma,port=20049,timeo=600,retrans=2, > sec=sys,mountaddr=192.168.2.55,mountvers=3,mountproto=tcp, > local_lock=none,addr=192.168.2.55 > > Linux morisot.1015granger.net 6.15.10-100.fc41.x86_64 #1 SMP > PREEMPT_DYNAMIC Fri Aug 15 14:55:12 UTC 2025 x86_64 GNU/Linux If you're using LOCALIO (client on server) that'd explain your not seeing any READs coming over the wire to NFSD. I've made sure to disable LOCALIO on my client, with: echo N > /sys/module/nfs/parameters/localio_enabled
