On 6/17/25 1:22 PM, Mike Snitzer wrote: > On Mon, Jun 16, 2025 at 12:10:38PM -0400, Mike Snitzer wrote: >> On Mon, Jun 16, 2025 at 09:32:16AM -0400, Chuck Lever wrote: >>> On 6/12/25 12:00 PM, Mike Snitzer wrote: >>>> On Thu, Jun 12, 2025 at 09:21:35AM -0400, Chuck Lever wrote: >>>>> On 6/11/25 3:18 PM, Mike Snitzer wrote: >>>>>> On Wed, Jun 11, 2025 at 10:31:20AM -0400, Chuck Lever wrote: >>>>>>> On 6/10/25 4:57 PM, Mike Snitzer wrote: >>>>>>>> Add 'enable-dontcache' to NFSD's debugfs interface so that: Any data >>>>>>>> read or written by NFSD will either not be cached (thanks to O_DIRECT) >>>>>>>> or will be removed from the page cache upon completion (DONTCACHE). >>>>>>> >>>>>>> I thought we were going to do two switches: One for reads and one for >>>>>>> writes? I could be misremembering. >>>>>> >>>>>> We did discuss the possibility of doing that. Still can-do if that's >>>>>> what you'd prefer. >>>>> >>>>> For our experimental interface, I think having read and write enablement >>>>> as separate settings is wise, so please do that. >>>>> >>>>> One quibble, though: The name "enable_dontcache" might be directly >>>>> meaningful to you, but I think others might find "enable_dont" to be >>>>> oxymoronic. And, it ties the setting to a specific kernel technology: >>>>> RWF_DONTCACHE. >>>>> >>>>> So: Can we call these settings "io_cache_read" and "io_cache_write" ? >>>>> >>>>> They could each carry multiple settings: >>>>> >>>>> 0: Use page cache >>>>> 1: Use RWF_DONTCACHE >>>>> 2: Use O_DIRECT >>>>> >>>>> You can choose to implement any or all of the above three mechanisms. >>>> >>>> I like it, will do for v2. But will have O_DIRECT=1 and RWF_DONTCACHE=2. >>> >>> For io_cache_read, either settings 1 and 2 need to set >>> disable_splice_read, or the io_cache_read setting has to be considered >>> by nfsd_read_splice_ok() when deciding to use nfsd_iter_read() or >>> splice read. >> >> Yes, I understand. >> >>> However, it would be slightly nicer if we could decide whether splice >>> read can be removed /before/ this series is merged. Can you get NFSD >>> tested with IOR with disable_splice_read both enabled and disabled (no >>> direct I/O)? Then we can compare the results to ensure that there is no >>> negative performance impact for removing the splice read code. >> >> I can ask if we have a small window of opportunity to get this tested, >> will let you know if so. >> > > I was able to enlist the help of Keith (cc'd) to get some runs in to > compare splice_read vs vectored read. A picture is worth 1000 words: > https://original.art/NFSD_splice_vs_buffered_read_IOR_EASY.jpg > > Left side is with splice_read running IOR_EASY with 48, 64, 96 PPN > (Processes Per Node on each client) respectively. Then the same > IOR_EASY workload progression for buffered IO on the right side. > > 6x servers with 1TB memory and 48 cpus, each configured with 32 NFSD > threads, with CPU pinning and 4M Read Ahead. 6x clients running IOR_EASY. > > This was Keith's take on splice_read's benefits: > - Is overall faster than buffered at any PPN. > - Is able to scale higher with PPN (whereas buffered is flat). > - Safe to say splice_read allows NFSD to do more IO then standard > buffered. I thank you and Keith for the data! > (These results came _after_ I did the patch to remove all the > splice_read related code from NFSD and SUNRPC.. while cathartic, alas > it seems it isn't meant to be at this point. I'll let you do the > honors in the future if/when you deem splice_read worthy of removal.) If we were to make all NFS READ operations use O_DIRECT, then of course NFSD's splice read should be removed at that point. -- Chuck Lever
