On 6/17/25 11:04 PM, hexue wrote: > The plug mechanism uses the merging of block I/O (bio) to reduce the > frequency of I/O submission to improve throughput. This mechanism can > greatly reduce the disk seek overhead of the HDD and plays a key role > in optimizing the speed of IO. However, with the improvement of > storage device speed, high-performance SSD combined with asynchronous > processing mechanisms such as io_uring has achieved very fast I/O > processing speed. The delay introduced by flow control and bio merging > may reduced the throughput to a certain extent. > > After testing, I found that plug increases the burden of high > concurrency of SSD on random IO and 128K sequential IO. But it still > has a certain optimization effect on small block (4k) sequential IO, > of course small sequential IO is the most suitable application for > merging scenarios, but the current plug does not distinguish between > different usage scenarios. > > I have made aggressive modifications to the kernel code to disable the > plug mechanism during I/O submission, the following are the random > performance differences after disabling only merging and completely > disabling plug (merging and flow control): > > ------------------------------------------------------------------------------------ > PCIe Gen4 SSD > 16GB Mem > Seq 128K > Random 4K > cmd: > taskset -c 0 ./t/io_uring -b 131072 -d128 -c32 -s32 -R0 -p1 -F1 -B1 -n1 -r5 /dev/nvme0n1 > taskset -c 0 ./t/io_uring -b 4096 -d128 -c32 -s32 -R1 -p1 -F1 -B1 -n1 -r5 /dev/nvme0n1 > data unit: IOPS > ------------------------------------------------------------------------------------ > Enable plug disable merge disable plug > Seq IO 50100 50133 50125 > Random IO 821K 824K 836K -1.83% > ------------------------------------------------------------------------------------ > > I used a higher-speed device (PCIe Gen5 server and PCIe Gen5 SSD) to verify the hypothesis > and found that the gap widened further. > > ------------------------------------------------------------------------------------ > Enable plug disable merge disable plug > Seq IO 88938 89832 89869 > Random IO 1.02M 1.022M 1.06M -3.92% > ------------------------------------------------------------------------------------ > > In the current kernel, there is a certain flag (REQ_NOMERGE_FLAGS) to > control whether IO operations can be merged. However, the decision for > plug selection is determined solely by whether batch submission is > enabled (state->need_plug = max_ios > 2;). I'm wondering whether this > judgment mechanism is still applicable to high-speed SSDs. 1M isn't really high speed, it's "normal" speed. When I did my previous testing, I used gen2 optane, which does 5M per device. But even flash based devices these days do 3M+ IOPS. > So the discussion points are: > - Will plugs gradually disappear as hardware devices develop? > - Is it reasonable to make flow control an optional > configuration? Or could we change the criteria for determining > when to apply plug? > - Are there other thoughts about plug that we can talk now? Those results are odd. For plugging, the main wins should be grabbing batches of tags and using ->queue_rqs for queueing the IO on the device side. In my past testing, those are a major win. It's also been a while since I've run peak testing, so it's also very possible that we've regressed there, unknowingly, and that's why you're not seeing any wins from plugging. For your test case, we should allocate 32 tags once, and then use those 32 tags for submission. That's a lot more efficient that allocating tags one-by-one as IO gets queued. And on the NVMe side, we should be submitting batches of 32 requests per doorbell write. Timestamp reductions are also generally a nice win. But without knowing what your profiles look like, it's impossible to say what's going on at your end. A lot more details on your runs would be required. Rather than pontificate on getting rid of plugging, I'd much rather see some investigation going into whether these optimizations are still happening as they should be. And if the answer is no, then what broken it? If the answer is yes, then why isn't it providing the speedup that it should. -- Jens Axboe
