Hello. On úterý 12. srpna 2025 2:45:02, středoevropský letní čas Damien Le Moal wrote: > On 8/12/25 5:42 AM, Oleksandr Natalenko wrote: > > On pondělí 11. srpna 2025 18:06:16, středoevropský letní čas David Rientjes wrote: > >> On Mon, 11 Aug 2025, Oleksandr Natalenko wrote: > >>> I'm fairly confident that the following commit > >>> > >>> 459779d04ae8d block: Improve read ahead size for rotational devices > >>> > >>> caused a regression in my test bench. > >>> > >>> I'm running v6.17-rc1 in a small QEMU VM with virtio-scsi disk. It has got 1 GiB of RAM, so I can saturate it easily causing reclaiming mechanism to kick in. > >>> > >>> If MGLRU is enabled: > >>> > >>> $ echo 1000 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms > >>> > >>> then, once page cache builds up, an OOM happens without reclaiming inactive file pages: [1]. Note that inactive_file:506952kB, I'd expect these to be reclaimed instead, like how it happens with v6.16. > >>> > >>> If MGLRU is disabled: > >>> > >>> $ echo 0 | sudo tee /sys/kernel/mm/lru_gen/min_ttl_ms > >>> > >>> then OOM doesn't occur, and things seem to work as usual. > >>> > >>> If MGLRU is enabled, and 459779d04ae8d is reverted on top of v6.17-rc1, the OOM doesn't happen either. > >>> > >>> Could you please check this? > >>> > >> > >> This looks to be an MGLRU policy decision rather than a readahead > >> regression, correct? > >> > >> Mem-Info: > >> active_anon:388 inactive_anon:5382 isolated_anon:0 > >> active_file:9638 inactive_file:126738 isolated_file:0 > >> > >> Setting min_ttl_ms to 1000 is preserving the working set and triggering > >> the oom kill is the only alternative to free memory in that configuration. > >> The oom kill is being triggered by kswapd for this purpose. > >> > >> So additional readahead would certainly increase that working set. This > >> looks working as intended. > > > > OK, this makes sense indeed, thanks for the explanation. But is inactive_file explosion expected and justified? > > > > Without revert: > > > > $ echo 3 | sudo tee /proc/sys/vm/drop_caches; free -m; sudo journalctl -kb >/dev/null; free -m > > 3 > > total used free shared buff/cache available > > Mem: 690 179 536 3 57 510 > > Swap: 1379 12 1367 > > /* OOM happens here */ > > total used free shared buff/cache available > > Mem: 690 177 52 3 561 513 > > Swap: 1379 17 1362 > > > > With revert: > > > > $ echo 3 | sudo tee /proc/sys/vm/drop_caches; free -m; sudo journalctl -kb >/dev/null; free -m > > 3 > > total used free shared buff/cache available > > Mem: 690 214 498 4 64 476 > > Swap: 1379 0 1379 > > /* no OOM */ > > total used free shared buff/cache available > > Mem: 690 209 462 4 119 481 > > Swap: 1379 0 1379 > > > > The journal folder size is: > > > > $ sudo du -hs /var/log/journal > > 575M /var/log/journal > > > > It looks like this readahead change causes far more data to be read than actually needed? > > For your drive as seen by the VM, what is the value of > /sys/block/sdX/queue/optimal_io_size ? > > I guess it is "0", as I see on my VM. Yes, it's 0. > So before 459779d04ae8d, the block device read_ahead_kb was 128KB only, and > 459779d04ae8d switched it to be 2 times the max_sectors_kb, so 8MB. This change > significantly improves file buffered read performance on HDDs, and HDDs only. Right, max_sectors_kb is 4096. > This means that your VM device is probably being reported as a rotational one > (/sys/block/sdX/queue/rotational is 1), which is normal if you attached an > actual HDD. If you are using a qcow2 image for that disk, then having > rotational==1 is questionable... Yes, it's reported as rotational by default. I've just set -device scsi-hd,drive=hd1,rotation_rate=1 so that guest will see the drive as non-rotational from now on, which brings old behaviour back. > The other issue is the device driver for the device reporting 0 for the optimal > IO size, which normally happens only for SATA drives. I see the same with > virtio-scsi, which is also questionable given that the maximum IO size with it > is fairly limited. So virtio-scsi may need some tweaking. > > The other thing to question, I think, is setting read_ahead_kb using the > optimal_io_size limit (io_opt), which can be *very large*. For most SCSI > devices, it is 16MB, so you will see a read_ahead_kb of 32 MB. But for SCSI > devices, optimal_io_size indicates a *maximum* IO size beyond which performance > may degrade. So using any value lower than this, but still reasonably large, > would be better in general I think. Note that lim->io_opt for RAID arrays > actually indicates the stripe size, so generally a lot smaller than the > component drives io_opt. And this use changes the meaning of that queue limit, > which makes things even more confusing and finding an adequate default harder. Thank you for the explanation. -- Oleksandr Natalenko, MSE
Attachment:
signature.asc
Description: This is a digitally signed message part.
