On Tue, Apr 22, 2025 at 10:13 PM Jens Axboe <axboe@xxxxxxxxx> wrote: > > On 4/22/25 8:10 AM, ??? wrote: > > On Tue, Apr 22, 2025 at 9:35?PM Jens Axboe <axboe@xxxxxxxxx> wrote: > >> > >> On 4/22/25 4:45 AM, Zhiwei Jiang wrote: > >>> In the Firecracker VM scenario, sporadically encountered threads with > >>> the UN state in the following call stack: > >>> [<0>] io_wq_put_and_exit+0xa1/0x210 > >>> [<0>] io_uring_clean_tctx+0x8e/0xd0 > >>> [<0>] io_uring_cancel_generic+0x19f/0x370 > >>> [<0>] __io_uring_cancel+0x14/0x20 > >>> [<0>] do_exit+0x17f/0x510 > >>> [<0>] do_group_exit+0x35/0x90 > >>> [<0>] get_signal+0x963/0x970 > >>> [<0>] arch_do_signal_or_restart+0x39/0x120 > >>> [<0>] syscall_exit_to_user_mode+0x206/0x260 > >>> [<0>] do_syscall_64+0x8d/0x170 > >>> [<0>] entry_SYSCALL_64_after_hwframe+0x78/0x80 > >>> The cause is a large number of IOU kernel threads saturating the CPU > >>> and not exiting. When the issue occurs, CPU usage 100% and can only > >>> be resolved by rebooting. Each thread's appears as follows: > >>> iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork_asm > >>> iou-wrk-44588 [kernel.kallsyms] [k] ret_from_fork > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_worker_handle_work > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_submit_work > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_issue_sqe > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_write > >>> iou-wrk-44588 [kernel.kallsyms] [k] blkdev_write_iter > >>> iou-wrk-44588 [kernel.kallsyms] [k] iomap_file_buffered_write > >>> iou-wrk-44588 [kernel.kallsyms] [k] iomap_write_iter > >>> iou-wrk-44588 [kernel.kallsyms] [k] fault_in_iov_iter_readable > >>> iou-wrk-44588 [kernel.kallsyms] [k] fault_in_readable > >>> iou-wrk-44588 [kernel.kallsyms] [k] asm_exc_page_fault > >>> iou-wrk-44588 [kernel.kallsyms] [k] exc_page_fault > >>> iou-wrk-44588 [kernel.kallsyms] [k] do_user_addr_fault > >>> iou-wrk-44588 [kernel.kallsyms] [k] handle_mm_fault > >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_fault > >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_no_page > >>> iou-wrk-44588 [kernel.kallsyms] [k] hugetlb_handle_userfault > >>> iou-wrk-44588 [kernel.kallsyms] [k] handle_userfault > >>> iou-wrk-44588 [kernel.kallsyms] [k] schedule > >>> iou-wrk-44588 [kernel.kallsyms] [k] __schedule > >>> iou-wrk-44588 [kernel.kallsyms] [k] __raw_spin_unlock_irq > >>> iou-wrk-44588 [kernel.kallsyms] [k] io_wq_worker_sleeping > >>> > >>> I tracked the address that triggered the fault and the related function > >>> graph, as well as the wake-up side of the user fault, and discovered this > >>> : In the IOU worker, when fault in a user space page, this space is > >>> associated with a userfault but does not sleep. This is because during > >>> scheduling, the judgment in the IOU worker context leads to early return. > >>> Meanwhile, the listener on the userfaultfd user side never performs a COPY > >>> to respond, causing the page table entry to remain empty. However, due to > >>> the early return, it does not sleep and wait to be awakened as in a normal > >>> user fault, thus continuously faulting at the same address,so CPU loop. > >>> Therefore, I believe it is necessary to specifically handle user faults by > >>> setting a new flag to allow schedule function to continue in such cases, > >>> make sure the thread to sleep. > >>> > >>> Patch 1 io_uring: Add new functions to handle user fault scenarios > >>> Patch 2 userfaultfd: Set the corresponding flag in IOU worker context > >>> > >>> fs/userfaultfd.c | 7 ++++++ > >>> io_uring/io-wq.c | 57 +++++++++++++++--------------------------------- > >>> io_uring/io-wq.h | 45 ++++++++++++++++++++++++++++++++++++-- > >>> 3 files changed, 68 insertions(+), 41 deletions(-) > >> > >> Do you have a test case for this? I don't think the proposed solution is > >> very elegant, userfaultfd should not need to know about thread workers. > >> I'll ponder this a bit... > >> > >> -- > >> Jens Axboe > > Sorry,The issue occurs very infrequently, and I can't manually > > reproduce it. It's not very elegant, but for corner cases, it seems > > necessary to make some compromises. > > I'm going to see if I can create one. Not sure I fully understand the > issue yet, but I'd be surprised if there isn't a more appropriate and > elegant solution rather than exposing the io-wq guts and having > userfaultfd manipulate them. That really should not be necessary. > > -- > Jens Axboe Thanks.I'm looking forward to your good news.
