Hi Rik!
Am 01.04.25 um 14:15 schrieb Rik Theys:
On 4/1/25 2:05 PM, Daniel Kobras wrote:
Am 15.12.24 um 13:38 schrieb Rik Theys:
Suddenly, a number of clients start to send an abnormal amount of NFS
traffic to the server that saturates their link and never seems to
stop. Running iotop on the clients shows kworker-
{rpciod,nfsiod,xprtiod} processes generating the write traffic. On
the server side, the system seems to process the traffic as the disks
are processing the write requests.
This behavior continues even after stopping all user processes on the
clients and unmounting the NFS mount on the client. Is this normal? I
was under the impression that once the NFS mount is unmounted no
further traffic to the server should be visible?
I'm currently looking at an issue that resembles your description
above (excess traffic to the server for data that was already written
and committed), and part of the packet capture also looks roughly
similar to what you've sent in a followup. Before I dig any deeper:
Did you manage to pinpoint or resolve the problem in the meantime?
Our server is currently running the 6.12 LTS kernel and we haven't had
this specific issue any more. But we were never able to reproduce it, so
unfortunately I can't say for sure if it's fixed, or what fixed it :-/.
Thanks for the update! Indeed, in the meantime the affected environment
here stopped showing the reported behavior as well after a few days, and
I don't have a clear indication what might have been the fix, either.
When the issue still occurred, it could (once) be provoked by dd'ing 4GB
of /dev/zero to a test file on an NFSv4.2 mount. The network trace shows
that the file is completely written at wire speed. But after a five
second pause, the client then starts sending the same file again in
smaller chunks of a few hundred MB at five second intervals. So it
appears that the file's pages are background-flushed to storage again,
even though they've already been written out. On the NFS layer, none of
the passes look conspicuous to me: WRITE and COMMIT operations all get
NFS4_OK'ed by the server.
Which kernel version(s) are your server and clients running?
The systems in the affected environment run Debian-packaged kernels. The
servers are on Debian's 6.1.0-32 which corresponds to upstream's
6.1.129. The issues was seen on clients running the same kernel version,
but also on older systems running Debian's 5.10.0-33, corresponding to
5.10.226 upstream. I've skimmed the list of patches that went into
either of these kernel versions, but nothing stood out as clearly related.
Kind regards,
Daniel
--
Daniel Kobras
Principal Architect
Puzzle ITC Deutschland
+49 7071 14316 0
www.puzzle-itc.de
--
Puzzle ITC Deutschland GmbH
Sitz der Gesellschaft: Eisenbahnstraße 1, 72072
Tübingen
Eingetragen am Amtsgericht Stuttgart HRB 765802
Geschäftsführer:
Lukas Kallies, Daniel Kobras, Mark Pröhl
