Hi,
On 5/16/25 7:51 AM, Rik Theys wrote:
Hi,
On 4/18/25 3:31 PM, Daniel Kobras wrote:
Hi Rik!
Am 01.04.25 um 14:15 schrieb Rik Theys:
On 4/1/25 2:05 PM, Daniel Kobras wrote:
Am 15.12.24 um 13:38 schrieb Rik Theys:
Suddenly, a number of clients start to send an abnormal amount of
NFS traffic to the server that saturates their link and never
seems to stop. Running iotop on the clients shows kworker-
{rpciod,nfsiod,xprtiod} processes generating the write traffic. On
the server side, the system seems to process the traffic as the
disks are processing the write requests.
This behavior continues even after stopping all user processes on
the clients and unmounting the NFS mount on the client. Is this
normal? I was under the impression that once the NFS mount is
unmounted no further traffic to the server should be visible?
I'm currently looking at an issue that resembles your description
above (excess traffic to the server for data that was already
written and committed), and part of the packet capture also looks
roughly similar to what you've sent in a followup. Before I dig any
deeper: Did you manage to pinpoint or resolve the problem in the
meantime?
Our server is currently running the 6.12 LTS kernel and we haven't
had this specific issue any more. But we were never able to
reproduce it, so unfortunately I can't say for sure if it's fixed,
or what fixed it :-/.
Thanks for the update! Indeed, in the meantime the affected
environment here stopped showing the reported behavior as well after
a few days, and I don't have a clear indication what might have been
the fix, either.
When the issue still occurred, it could (once) be provoked by dd'ing
4GB of /dev/zero to a test file on an NFSv4.2 mount. The network
trace shows that the file is completely written at wire speed. But
after a five second pause, the client then starts sending the same
file again in smaller chunks of a few hundred MB at five second
intervals. So it appears that the file's pages are background-flushed
to storage again, even though they've already been written out. On
the NFS layer, none of the passes look conspicuous to me: WRITE and
COMMIT operations all get NFS4_OK'ed by the server.
Which kernel version(s) are your server and clients running?
The systems in the affected environment run Debian-packaged kernels.
The servers are on Debian's 6.1.0-32 which corresponds to upstream's
6.1.129. The issues was seen on clients running the same kernel
version, but also on older systems running Debian's 5.10.0-33,
corresponding to 5.10.226 upstream. I've skimmed the list of patches
that went into either of these kernel versions, but nothing stood out
as clearly related.
Our server and clients are currently showing the same behavior again:
clients are sending abnormal amounts of write traffic to the NFS
server and the server is actually processing it as the writes end up
on the disk (which fills up our replication journals). iotop shows
that the kworker-{rpciod,nfsiod,xprtiod} are responsible for this
traffic. A reboot of the server does not solve the issue. Also
rebooting individual clients that are participating in this does not
help. After a few minutes of user traffic they show the same behavior
again. We also see this on multiple clients at the same time.
The NFS operations that are being sent are mostly putfh, sequence and
getattr.
The server is running upstream 6.12.25 and the clients are running
Rocky 8 (4.18.0-553.51.1.el8_10) and 9 (5.14.0-503.38.1.el9_5).
What are some of the steps we can take to debug the root cause of
this? Any idea on how to stop this traffic flood?
I took a tcpdump on one of the clients that was doing this. The pcap was
stored on the local disk of the server. When I tried to copy the pcap to
our management server over scp it now hangs at 95%. The target disk on
the management server is also an NFS mount of the affected server. The
scp had copied 565MB and our management server has now also started to
flood the server with non-stop traffic (basically saturating its link).
The management server is running Debian's 6.1.135 kernel.
It seems that once a client has triggered some bad state in the server,
other clients that write a large file to the server also start to
participate in this behavior. Rebooting the server does not seem to help
as the same state is triggered almost immediately again by some client.
Regards,
Rik
--
Rik Theys
System Engineer
KU Leuven - Dept. Elektrotechniek (ESAT)
Kasteelpark Arenberg 10 bus 2440 - B-3001 Leuven-Heverlee
+32(0)16/32.11.07
----------------------------------------------------------------
<<Any errors in spelling, tact or fact are transmission errors>>
