On 2025-05-29 15:33:47 -0700, Mohamed Khalfella wrote: > On 2025-05-29 16:13:23 -0600, Keith Busch wrote: > > On Thu, May 29, 2025 at 03:49:28PM -0600, Mohamed Khalfella wrote: > > > nvme-fc initiator hit hung_task with stacktrace above while handling > > > request timeout call. The work thread is waiting for itself to finish > > > which is never going to happen. From the stacktrace the nvme controller > > > was in NVME_CTRL_CONNECTING state when nvme_fc_timeout() was called. > > > We do not expect to get IO timeout call in NVME_CTRL_CONNECTING state > > > because blk_sync_queue() must have been called on this queue before > > > switching from NVME_CTRL_RESETTING to NVME_CTRL_CONNECTING. > > > > > > It turned out that blk_sync_queue() did not stop q->timeout_work from > > > running as expected. nvme_fc_timeout() returned BLK_EH_RESET_TIMER > > > causing q->timeout to be rearmed after it was canceled earlier. > > > q->timeout queued q->timeout_work after the controller switched to > > > NVME_CTRL_CONNECTING state causing deadlock above. > > > > > > Add QUEUE_FLAG_NOTIMEOUT queue flag to tell q->timeout not to queue > > > q->timeout_work while queue is being synced. Update blk_sync_queue() to > > > cancel q->timeout_work first and then cancel q->timeout. > > > > I feel like this is a nvme-fc problem that doesn't need the block layer > > to handle. Just don't sync the queues within the timeout workqueue > > context. > > Agreed on nvme-fc should not sync queues within timeout work, and I am > testing a patch to fix nvme-fc. At the same time blk_sync_queue() should > provide a guarantee that q->timeout_work will not run after the function > returns, no? Following up on this patch. I think the issue with blk_sync_queue() needs to be addressed. If adding a queue flag is not the preferred way to do it, please let me know what do you suggest and I will make the code changes.
