Currently ublk zero-copy requires ublk request buffers to be registered and unregistered by the ublk I/O's daemon task. However, as currently implemented, there is no reason for this restriction. Registration looks up the request via the ublk device's tagset rather than the daemon-local ublk_io structure and takes an atomic reference to prevent racing with dispatch or completion of the request. Ming has expressed interest in relaxing this restriction[1] so the ublk server can offload the I/O operation that uses the zero-copy buffer to another thread. Additionally, optimize the buffer registration for the common case where the buffer is registered and unregistered by the daemon task. Skip the expensive atomic reference count increment and decrement and the several pointer dereferences involved in looking up the request on the tagset. On our ublk server threads, this results in a 24% decrease in the CPU time spent in the ublk functions on a 4K zero-copy read workload, from 8.75% to 6.68%. Two preliminary fixes are included as well: - Move the ublk request reference count from ublk_rq_data (a tail allocation of struct request) to ublk_io to prevent a use after free if the struct request is freed concurrently with taking a reference. - Don't allocate physically contiguous memory for __queues, which can be very large. Doubling the size of struct ublk_io caused ENOMEM errors when adding a ublk device before this change. [1]: https://lore.kernel.org/linux-block/aAmYJxaV1-yWEMRo@fedora/ v2: - Add 2 fix patches - Optimize buffer unregistration too - Cache-align ublk_io - Restore check for zero copy support in ublk_unregister_io_buf() - Check for registered buffer count overflow Caleb Sander Mateos (14): ublk: use vmalloc for ublk_device's __queues ublk: remove struct ublk_rq_data ublk: check cmd_op first ublk: handle UBLK_IO_FETCH_REQ earlier ublk: remove task variable from __ublk_ch_uring_cmd() ublk: consolidate UBLK_IO_FLAG_{ACTIVE,OWNED_BY_SRV} checks ublk: move ublk_prep_cancel() to case UBLK_IO_COMMIT_AND_FETCH_REQ ublk: don't take ublk_queue in ublk_unregister_io_buf() ublk: allow UBLK_IO_(UN)REGISTER_IO_BUF on any task ublk: return early if blk_should_fake_timeout() ublk: optimize UBLK_IO_REGISTER_IO_BUF on daemon task ublk: optimize UBLK_IO_UNREGISTER_IO_BUF on daemon task ublk: remove ubq checks from ublk_{get,put}_req_ref() ublk: cache-align struct ublk_io drivers/block/ublk_drv.c | 283 +++++++++++++++++++++------------- include/uapi/linux/ublk_cmd.h | 10 ++ 2 files changed, 187 insertions(+), 106 deletions(-) -- 2.45.2
