[no subject]

There has been a few improvements though during Linux 6.9 on the log performance,
but I can't tell if you have any of those improvements around.

I'd suggest you trying to run a newer upstream kernel, otherwise you'll get very
limited support from the upstream community. If you can't, I'd suggest you
reporting this issue to your vendor, so they can track what you are/are not
using in your current kernel.

FWIW, I'm not sure if NixOS uses linux-stable kernels or not. If that's the
case, running a newer kernel suggestion is still valid.

Carlos

> 
> 
> 
> Regards,
> Christian
> 
> --
> Christian Theune · ct@xxxxxxxxxxxxxxx · +49 345 219401 0
> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
> 

> Jun 10 16:03:08 machine1 kernel: INFO: task xfsaild/vda1:188 blocked for more than 122 seconds.
> Jun 10 16:03:08 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:08 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:08 machine1 kernel: task:xfsaild/vda1    state:D stack:0     pid:188   ppid:2      flags:0x00004000
> Jun 10 16:03:08 machine1 kernel: Call Trace:
> Jun 10 16:03:08 machine1 kernel:  <TASK>
> Jun 10 16:03:08 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:08 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:08 machine1 kernel:  schedule_timeout+0x159/0x170
> Jun 10 16:03:08 machine1 kernel:  wait_for_completion+0x8a/0x160
> Jun 10 16:03:08 machine1 kernel:  __flush_workqueue+0x157/0x440
> Jun 10 16:03:08 machine1 kernel:  ? finish_task_switch.isra.0+0x94/0x300
> Jun 10 16:03:08 machine1 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 10 16:03:08 machine1 kernel:  ? __timer_delete_sync+0x7d/0xe0
> Jun 10 16:03:08 machine1 kernel:  xfs_log_force+0x7e/0x240 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfsaild+0x17d/0x870 [xfs]
> Jun 10 16:03:08 machine1 kernel:  ? __set_cpus_allowed_ptr_locked+0xf4/0x1c0
> Jun 10 16:03:08 machine1 kernel:  ? __pfx_xfsaild+0x10/0x10 [xfs]
> Jun 10 16:03:08 machine1 kernel:  kthread+0xe8/0x120
> Jun 10 16:03:08 machine1 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 10 16:03:08 machine1 kernel:  ret_from_fork+0x34/0x50
> Jun 10 16:03:08 machine1 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 10 16:03:08 machine1 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 10 16:03:08 machine1 kernel:  </TASK>
> Jun 10 16:03:08 machine1 kernel: INFO: task systemd-journal:407 blocked for more than 122 seconds.
> Jun 10 16:03:08 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:08 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:08 machine1 kernel: task:systemd-journal state:D stack:0     pid:407   ppid:1      flags:0x00000002
> Jun 10 16:03:08 machine1 kernel: Call Trace:
> Jun 10 16:03:08 machine1 kernel:  <TASK>
> Jun 10 16:03:08 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:08 machine1 kernel:  ? __mod_memcg_lruvec_state+0x58/0xb0
> Jun 10 16:03:08 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:08 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:08 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:08 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:08 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:08 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:08 machine1 kernel:  ? filename_lookup+0xe8/0x200
> Jun 10 16:03:08 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:08 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:08 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:08 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:08 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:08 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:08 machine1 kernel: RIP: 0033:0x7fc7798def5f
> Jun 10 16:03:08 machine1 kernel: RSP: 002b:00007ffd050a0ee0 EFLAGS: 00010246
> Jun 10 16:03:08 machine1 kernel: RAX: 00007fc767892d68 RBX: 000055d3034cd900 RCX: 00007fc77999d300
> Jun 10 16:03:08 machine1 kernel: RDX: 0000000000000000 RSI: 00007fc767892d68 RDI: 000055d3034cd900
> Jun 10 16:03:08 machine1 kernel: RBP: 0000000002bc3c40 R08: 00000000000000c9 R09: 000055d3034cd938
> Jun 10 16:03:08 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007ffd050a1b00
> Jun 10 16:03:08 machine1 kernel: R13: 00000000001df770 R14: 00007ffd050a10a8 R15: 00007ffd050a10a0
> Jun 10 16:03:08 machine1 kernel:  </TASK>
> Jun 10 16:03:08 machine1 kernel: INFO: task systemd-timesyn:804 blocked for more than 122 seconds.
> Jun 10 16:03:08 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:08 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:08 machine1 kernel: task:systemd-timesyn state:D stack:0     pid:804   ppid:1      flags:0x00000002
> Jun 10 16:03:08 machine1 kernel: Call Trace:
> Jun 10 16:03:08 machine1 kernel:  <TASK>
> Jun 10 16:03:08 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:08 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:08 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_trans_alloc_ichange+0x85/0x200 [xfs]
> Jun 10 16:03:08 machine1 kernel:  xfs_setattr_nonsize+0x9a/0x390 [xfs]
> Jun 10 16:03:08 machine1 kernel:  notify_change+0x1f5/0x4c0
> Jun 10 16:03:08 machine1 kernel:  ? vfs_utimes+0x141/0x270
> Jun 10 16:03:08 machine1 kernel:  vfs_utimes+0x141/0x270
> Jun 10 16:03:08 machine1 kernel:  do_utimes+0xf8/0x160
> Jun 10 16:03:08 machine1 kernel:  __x64_sys_utimensat+0xa1/0xf0
> Jun 10 16:03:08 machine1 kernel:  do_syscall_64+0x39/0x90
> Jun 10 16:03:08 machine1 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 10 16:03:08 machine1 kernel: RIP: 0033:0x7f0c14f0ae53
> Jun 10 16:03:08 machine1 kernel: RSP: 002b:00007fff88b9d388 EFLAGS: 00000206 ORIG_RAX: 0000000000000118
> Jun 10 16:03:08 machine1 kernel: RAX: ffffffffffffffda RBX: 0000000000000011 RCX: 00007f0c14f0ae53
> Jun 10 16:03:08 machine1 kernel: RDX: 0000000000000000 RSI: 00007f0c15320f09 RDI: 0000000000000011
> Jun 10 16:03:08 machine1 kernel: RBP: 0000000000000000 R08: 00000000ffffffff R09: 00000000ffffffff
> Jun 10 16:03:08 machine1 kernel: R10: 0000000000001000 R11: 0000000000000206 R12: 000055aa6cdbc000
> Jun 10 16:03:08 machine1 kernel: R13: 00000000ffffffff R14: ffffffffffffffff R15: 00000000ffffffff
> Jun 10 16:03:08 machine1 kernel:  </TASK>
> Jun 10 16:03:08 machine1 kernel: INFO: task filebeat:1200 blocked for more than 122 seconds.
> Jun 10 16:03:08 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:08 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:08 machine1 kernel: task:filebeat        state:D stack:0     pid:1200  ppid:1      flags:0x00000002
> Jun 10 16:03:08 machine1 kernel: Call Trace:
> Jun 10 16:03:08 machine1 kernel:  <TASK>
> Jun 10 16:03:08 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:08 machine1 kernel:  ? ip6_input_finish+0x48/0x70
> Jun 10 16:03:11 machine1 kernel:  ? _copy_to_iter+0x62/0x4b0
> Jun 10 16:03:11 machine1 kernel:  ? free_unref_page_commit+0x76/0x180
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  kiocb_modified+0x9b/0xd0
> Jun 10 16:03:11 machine1 kernel:  xfs_file_write_checks+0x1dd/0x280 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_file_buffered_write+0x60/0x2b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  ? selinux_file_permission+0x119/0x160
> Jun 10 16:03:11 machine1 kernel:  vfs_write+0x24f/0x430
> Jun 10 16:03:11 machine1 kernel:  ksys_write+0x6f/0xf0
> Jun 10 16:03:11 machine1 kernel:  do_syscall_64+0x39/0x90
> Jun 10 16:03:11 machine1 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x116ac4e
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:000000c000925600 EFLAGS: 00000212 ORIG_RAX: 0000000000000001
> Jun 10 16:03:11 machine1 kernel: RAX: ffffffffffffffda RBX: 0000000000000006 RCX: 000000000116ac4e
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000175 RSI: 000000c000602000 RDI: 0000000000000006
> Jun 10 16:03:11 machine1 kernel: RBP: 000000c000925640 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000212 R12: 000000c000925770
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000000010 R14: 000000c00044ea80 R15: 000000c00066c7e0
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:1168 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:1168  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe565fd2a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe4811ddd0 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d080
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:1253 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:1253  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe47ffe2a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe40013590 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d100
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2783 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2783  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe44df92a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe38132790 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d180
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2787 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2787  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  ? raw6_local_deliver+0x12a/0x250
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe3d0fa2a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe34013c80 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d1c0
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2791 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2791  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe2ddea250 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe28203aa0 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d240
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2792 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2792  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe2d5fa2a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe34135cb0 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d2c0
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Killing process 407 (systemd-journal) with signal SIGABRT.
> Jun 10 16:03:11 machine1 systemd[1]: Starting Serial console liveness marker...
> Jun 10 16:03:11 machine1 systemd[1]: Starting Logrotate Service...
> Jun 10 16:03:11 machine1 systemd[1]: serial-console-liveness.service: Deactivated successfully.
> Jun 10 16:03:11 machine1 systemd[1]: Finished Serial console liveness marker.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Process 407 (systemd-journal) of user 0 terminated abnormally with signal 6/ABRT, processing...
> Jun 10 16:03:11 machine1 systemd[1]: logrotate.service: Deactivated successfully.
> Jun 10 16:03:11 machine1 systemd[1]: Finished Logrotate Service.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Process 407 (systemd-journal) of user 0 dumped core.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.42b57193ad0b49a48e0dd3e3abe28b31.407.1749564188000000.zst
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libzstd.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libaudit.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libattr.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libseccomp.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libpam.so.0 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libcrypt.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libcap.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libacl.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Stack trace of thread 407:
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #0  0x00007fc77968b4c0 n/a (libsystemd-shared-256.so + 0x8b4c0)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #1  0x00007fc7798decb4 journal_file_append_data (libsystemd-shared-256.so + 0x2decb4)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #2  0x00007fc7798df3f5 journal_file_append_entry (libsystemd-shared-256.so + 0x2df3f5)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #3  0x000055d2da095ff3 server_dispatch_message_real (systemd-journald + 0x11ff3)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #4  0x000055d2da08e6a0 dev_kmsg_record (systemd-journald + 0xa6a0)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #5  0x000055d2da08ec53 server_read_dev_kmsg (systemd-journald + 0xac53)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #6  0x00007fc77990b223 source_dispatch (libsystemd-shared-256.so + 0x30b223)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #7  0x00007fc77990b518 sd_event_dispatch (libsystemd-shared-256.so + 0x30b518)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #8  0x00007fc77990c040 sd_event_run (libsystemd-shared-256.so + 0x30c040)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #9  0x000055d2da08d60a main (systemd-journald + 0x960a)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #10 0x00007fc77943227e __libc_start_call_main (libc.so.6 + 0x2a27e)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #11 0x00007fc779432339 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a339)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #12 0x000055d2da08db45 _start (systemd-journald + 0x9b45)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Stack trace of thread 274174:
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #0  0x00007fc77950d6ea fsync (libc.so.6 + 0x1056ea)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #1  0x00007fc779739519 journal_file_set_offline_internal (libsystemd-shared-256.so + 0x139519)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #2  0x00007fc779739880 journal_file_set_offline_thread (libsystemd-shared-256.so + 0x139880)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #3  0x00007fc779498af3 start_thread (libc.so.6 + 0x90af3)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #4  0x00007fc779517d34 __clone (libc.so.6 + 0x10fd34)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: ELF object binary architecture: AMD x86-64
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Consumed 2min 32.716s CPU time, 1.1G memory peak, 28.3M read from disk, 53.8G written to disk.
> Jun 10 16:03:11 machine1 systemd[1]: run-credentials-systemd\x2djournald.service.mount: Deactivated successfully.
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
> Jun 10 16:03:11 machine1 systemd[1]: Starting Journal Service...
> Jun 10 16:03:11 machine1 systemd-journald[274205]: Collecting audit messages is enabled.
> Jun 10 16:03:11 machine1 systemd-journald[274205]: File /var/log/journal/04256dcb074d45a997c6639bd2b566bb/system.journal corrupted or uncleanly shut down, renaming and replacing.
> Jun 10 16:03:11 machine1 systemd[1]: Started Journal Service.
> Jun 10 16:03:11 machine1 systemd-journald[274205]: File /var/log/journal/04256dcb074d45a997c6639bd2b566bb/user-31004.journal corrupted or uncleanly shut down, renaming and replacing.
> Jun 10 16:04:32 machine1 systemd[1]: fc-agent.service: Deactivated successfully.
> Jun 10 16:04:32 machine1 systemd[1]: Finished Flying Circus Management Task.
> Jun 10 16:04:32 machine1 systemd[1]: fc-agent.service: Consumed 25.773s CPU time, 1.6G memory peak, 18.5M read from disk, 479.4M written to disk, 125.2K incoming IP traffic, 20.8K o>
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d1c0
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2791 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2791  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe2ddea250 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe28203aa0 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d240
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: INFO: task slapd:2792 blocked for more than 122 seconds.
> Jun 10 16:03:11 machine1 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 10 16:03:11 machine1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 10 16:03:11 machine1 kernel: task:slapd           state:D stack:0     pid:2792  ppid:1      flags:0x00000002
> Jun 10 16:03:11 machine1 kernel: Call Trace:
> Jun 10 16:03:11 machine1 kernel:  <TASK>
> Jun 10 16:03:11 machine1 kernel:  __schedule+0x3fc/0x1440
> Jun 10 16:03:11 machine1 kernel:  schedule+0x5e/0xe0
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_wait+0xba/0x220 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xlog_grant_head_check+0xdf/0x110 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_log_reserve+0xc4/0x1f0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_reserve+0x138/0x170 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 10 16:03:11 machine1 kernel:  xfs_vn_update_time+0x90/0x1b0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  file_update_time+0x61/0x90
> Jun 10 16:03:11 machine1 kernel:  __xfs_filemap_fault.constprop.0+0xd4/0x1d0 [xfs]
> Jun 10 16:03:11 machine1 kernel:  do_page_mkwrite+0x50/0xb0
> Jun 10 16:03:11 machine1 kernel:  do_wp_page+0x431/0xb80
> Jun 10 16:03:11 machine1 kernel:  ? __pte_offset_map+0x1b/0x190
> Jun 10 16:03:11 machine1 kernel:  __handle_mm_fault+0xbcf/0xd90
> Jun 10 16:03:11 machine1 kernel:  handle_mm_fault+0x17f/0x370
> Jun 10 16:03:11 machine1 kernel:  do_user_addr_fault+0x1ee/0x630
> Jun 10 16:03:11 machine1 kernel:  exc_page_fault+0x71/0x160
> Jun 10 16:03:11 machine1 kernel:  asm_exc_page_fault+0x26/0x30
> Jun 10 16:03:11 machine1 kernel: RIP: 0033:0x4d0c04
> Jun 10 16:03:11 machine1 kernel: RSP: 002b:00007efe2d5fa2a0 EFLAGS: 00010246
> Jun 10 16:03:11 machine1 kernel: RAX: 000000000000a50b RBX: 00007efe34135cb0 RCX: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: RBP: 0000000039dfec40 R08: 0000000000000000 R09: 0000000000000000
> Jun 10 16:03:11 machine1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007efe9377d2c0
> Jun 10 16:03:11 machine1 kernel: R13: 0000000000020000 R14: 000000000000048b R15: 00007efe9377d000
> Jun 10 16:03:11 machine1 kernel:  </TASK>
> Jun 10 16:03:11 machine1 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
> Jun 10 16:03:11 machine1 systemd[1]: systemd-journald.service: Killing process 407 (systemd-journal) with signal SIGABRT.
> Jun 10 16:03:11 machine1 systemd[1]: Starting Serial console liveness marker...
> Jun 10 16:03:11 machine1 systemd[1]: Starting Logrotate Service...
> Jun 10 16:03:11 machine1 systemd[1]: serial-console-liveness.service: Deactivated successfully.
> Jun 10 16:03:11 machine1 systemd[1]: Finished Serial console liveness marker.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Process 407 (systemd-journal) of user 0 terminated abnormally with signal 6/ABRT, processing...
> Jun 10 16:03:11 machine1 systemd[1]: logrotate.service: Deactivated successfully.
> Jun 10 16:03:11 machine1 systemd[1]: Finished Logrotate Service.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Process 407 (systemd-journal) of user 0 dumped core.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.42b57193ad0b49a48e0dd3e3abe28b31.407.1749564188000000.zst
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libzstd.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libaudit.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libattr.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libseccomp.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libpam.so.0 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libcrypt.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libcap.so.2 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Module libacl.so.1 without build-id.
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Stack trace of thread 407:
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #0  0x00007fc77968b4c0 n/a (libsystemd-shared-256.so + 0x8b4c0)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #1  0x00007fc7798decb4 journal_file_append_data (libsystemd-shared-256.so + 0x2decb4)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #2  0x00007fc7798df3f5 journal_file_append_entry (libsystemd-shared-256.so + 0x2df3f5)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #3  0x000055d2da095ff3 server_dispatch_message_real (systemd-journald + 0x11ff3)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #4  0x000055d2da08e6a0 dev_kmsg_record (systemd-journald + 0xa6a0)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #5  0x000055d2da08ec53 server_read_dev_kmsg (systemd-journald + 0xac53)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #6  0x00007fc77990b223 source_dispatch (libsystemd-shared-256.so + 0x30b223)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #7  0x00007fc77990b518 sd_event_dispatch (libsystemd-shared-256.so + 0x30b518)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #8  0x00007fc77990c040 sd_event_run (libsystemd-shared-256.so + 0x30c040)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #9  0x000055d2da08d60a main (systemd-journald + 0x960a)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #10 0x00007fc77943227e __libc_start_call_main (libc.so.6 + 0x2a27e)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #11 0x00007fc779432339 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a339)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #12 0x000055d2da08db45 _start (systemd-journald + 0x9b45)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: Stack trace of thread 274174:
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #0  0x00007fc77950d6ea fsync (libc.so.6 + 0x1056ea)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #1  0x00007fc779739519 journal_file_set_offline_internal (libsystemd-shared-256.so + 0x139519)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #2  0x00007fc779739880 journal_file_set_offline_thread (libsystemd-shared-256.so + 0x139880)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #3  0x00007fc779498af3 start_thread (libc.so.6 + 0x90af3)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: #4  0x00007fc779517d34 __clone (libc.so.6 + 0x10fd34)
> Jun 10 16:03:11 machine1 systemd-coredump[274177]: ELF object binary architecture: AMD x86-64

> Jun 11 15:07:13 machine2 kernel: INFO: task kcompactd0:45 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:kcompactd0      state:D stack:0     pid:45    ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  io_schedule+0x46/0x80
> Jun 11 15:07:13 machine2 kernel:  folio_wait_bit_common+0x13d/0x360
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_wake_page_function+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  migrate_pages_batch+0x5c9/0xb90
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_compaction_free+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_compaction_alloc+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_remove_migration_pte+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  migrate_pages+0xbdc/0xd50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_compaction_alloc+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_compaction_free+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  compact_zone+0x831/0xf30
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  proactive_compact_node+0x85/0xe0
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  kcompactd+0x387/0x460
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kcompactd+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task xfsaild/vda1:221 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:xfsaild/vda1    state:D stack:0     pid:221   ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_timeout+0x159/0x170
> Jun 11 15:07:13 machine2 kernel:  wait_for_completion+0x8a/0x160
> Jun 11 15:07:13 machine2 kernel:  __flush_workqueue+0x157/0x440
> Jun 11 15:07:13 machine2 kernel:  ? finish_task_switch.isra.0+0x94/0x300
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? __timer_delete_sync+0x7d/0xe0
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force+0x7e/0x240 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfsaild+0x17d/0x870 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_xfsaild+0x10/0x10 [xfs]
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task journal-offline:804376 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:journal-offline state:D stack:0     pid:804376 ppid:1      flags:0x00004002
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? psi_task_switch+0xb7/0x200
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_timeout+0x159/0x170
> Jun 11 15:07:13 machine2 kernel:  wait_for_completion+0x8a/0x160
> Jun 11 15:07:13 machine2 kernel:  __flush_workqueue+0x157/0x440
> Jun 11 15:07:13 machine2 kernel:  ? __folio_batch_release+0x1f/0x60
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force_seq+0x75/0x180 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  __x64_sys_fsync+0x3b/0x70
> Jun 11 15:07:13 machine2 kernel:  do_syscall_64+0x39/0x90
> Jun 11 15:07:13 machine2 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 11 15:07:13 machine2 kernel: RIP: 0033:0x7f208f90d6ea
> Jun 11 15:07:13 machine2 kernel: RSP: 002b:00007f208effed70 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
> Jun 11 15:07:13 machine2 kernel: RAX: ffffffffffffffda RBX: 00005573438972c0 RCX: 00007f208f90d6ea
> Jun 11 15:07:13 machine2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000020
> Jun 11 15:07:13 machine2 kernel: RBP: 00007f208fd77700 R08: 0000000000000000 R09: 00007f208efff6c0
> Jun 11 15:07:13 machine2 kernel: R10: 00007f208f898886 R11: 0000000000000246 R12: fffffffffffffe90
> Jun 11 15:07:13 machine2 kernel: R13: 0000000000000000 R14: 00007fff9b527620 R15: 00007f208fef3000
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task journal-offline:804377 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:journal-offline state:D stack:0     pid:804377 ppid:1      flags:0x00004002
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? psi_task_switch+0xb7/0x200
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_timeout+0x159/0x170
> Jun 11 15:07:13 machine2 kernel:  wait_for_completion+0x8a/0x160
> Jun 11 15:07:13 machine2 kernel:  __flush_workqueue+0x157/0x440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force_seq+0x75/0x180 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  __x64_sys_fsync+0x3b/0x70
> Jun 11 15:07:13 machine2 kernel:  do_syscall_64+0x39/0x90
> Jun 11 15:07:13 machine2 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 11 15:07:13 machine2 kernel: RIP: 0033:0x7f208f90d6ea
> Jun 11 15:07:13 machine2 kernel: RSP: 002b:00007f20847fed70 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
> Jun 11 15:07:13 machine2 kernel: RAX: ffffffffffffffda RBX: 0000557343889080 RCX: 00007f208f90d6ea
> Jun 11 15:07:13 machine2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000002b
> Jun 11 15:07:13 machine2 kernel: RBP: 00007f208fd77700 R08: 0000000000000000 R09: 00007f20847ff6c0
> Jun 11 15:07:13 machine2 kernel: R10: 00007f208f898886 R11: 0000000000000246 R12: fffffffffffffe90
> Jun 11 15:07:13 machine2 kernel: R13: 0000000000000000 R14: 00007fff9b527620 R15: 00007f208fef3000
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task kworker/u9:0:795821 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:kworker/u9:0    state:D stack:0     pid:795821 ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? kick_pool+0x5a/0xf0
> Jun 11 15:07:13 machine2 kernel:  ? __queue_work.part.0+0x1a5/0x390
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  xlog_state_get_iclog_space+0x107/0x2c0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  xlog_write+0x7a/0x470 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? merge+0x42/0xc0
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? xlog_cil_order_write+0x9a/0x170 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_xlog_cil_order_cmp+0x10/0x10 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_work+0x72c/0x8f0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  process_one_work+0x18d/0x3a0
> Jun 11 15:07:13 machine2 kernel:  worker_thread+0x28c/0x3b0
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task kworker/2:0:800877 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:kworker/2:0     state:D stack:0     pid:800877 ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Workqueue: xfs-sync/vda1 xfs_log_worker [xfs]
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? ttwu_do_activate+0x64/0x230
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_timeout+0x159/0x170
> Jun 11 15:07:13 machine2 kernel:  wait_for_completion+0x8a/0x160
> Jun 11 15:07:13 machine2 kernel:  __flush_workqueue+0x157/0x440
> Jun 11 15:07:13 machine2 kernel:  ? update_load_avg+0x7e/0x7b0
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? kfree_rcu_monitor+0x32c/0x550
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force+0x7e/0x240 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_log_worker+0x3f/0xd0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  process_one_work+0x18d/0x3a0
> Jun 11 15:07:13 machine2 kernel:  worker_thread+0x28c/0x3b0
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task kworker/u9:1:802505 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:kworker/u9:1    state:D stack:0     pid:802505 ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Workqueue: writeback wb_workfn (flush-253:0)
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_timeout+0x159/0x170
> Jun 11 15:07:13 machine2 kernel:  wait_for_completion+0x8a/0x160
> Jun 11 15:07:13 machine2 kernel:  __flush_workqueue+0x157/0x440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_now.isra.0+0x5e/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_force_seq+0x69/0x250 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? xfs_btree_insert+0x10f/0x290 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force+0x7e/0x240 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_buf_lock+0xf2/0x100 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_buf_find_lock+0x5b/0x120 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_buf_get_map+0x1ab/0xa10 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_trans_get_buf_map+0x11e/0x1d0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_free_agfl_block+0xcd/0x100 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  xfs_agfl_free_finish_item+0x176/0x190 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_defer_finish_noroll+0x196/0x720 [xfs]
> Jun 11 15:07:13 machine2 kernel:  __xfs_trans_commit+0x1f0/0x3e0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_bmapi_convert_one_delalloc+0x351/0x490 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_bmapi_convert_delalloc+0x43/0x60 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_map_blocks+0x25b/0x430 [xfs]
> Jun 11 15:07:13 machine2 kernel:  iomap_do_writepage+0x23a/0x880
> Jun 11 15:07:13 machine2 kernel:  write_cache_pages+0x142/0x3b0
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_iomap_do_writepage+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ? throtl_add_bio_tg+0x35/0x90
> Jun 11 15:07:13 machine2 kernel:  iomap_writepages+0x20/0x50
> Jun 11 15:07:13 machine2 kernel:  xfs_vm_writepages+0x67/0xa0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  do_writepages+0xcf/0x1f0
> Jun 11 15:07:13 machine2 kernel:  __writeback_single_inode+0x3d/0x370
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  writeback_sb_inodes+0x1f5/0x4c0
> Jun 11 15:07:13 machine2 kernel:  __writeback_inodes_wb+0x4c/0xf0
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  wb_writeback+0x2d6/0x340
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  wb_workfn+0x35b/0x520
> Jun 11 15:07:13 machine2 kernel:  ? __schedule+0x404/0x1440
> Jun 11 15:07:13 machine2 kernel:  process_one_work+0x18d/0x3a0
> Jun 11 15:07:13 machine2 kernel:  worker_thread+0x28c/0x3b0
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task kworker/u9:3:803550 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:kworker/u9:3    state:D stack:0     pid:803550 ppid:2      flags:0x00004000
> Jun 11 15:07:13 machine2 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_order_write+0x134/0x170 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  xlog_cil_push_work+0x5ab/0x8f0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  process_one_work+0x18d/0x3a0
> Jun 11 15:07:13 machine2 kernel:  worker_thread+0x28c/0x3b0
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  kthread+0xe8/0x120
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork+0x34/0x50
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task nix-build:803925 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:nix-build       state:D stack:0     pid:803925 ppid:803923 flags:0x00004002
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? srso_return_thunk+0x5/0x5f
> Jun 11 15:07:13 machine2 kernel:  ? submit_bio_noacct+0x256/0x5a0
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  xlog_wait_on_iclog+0x174/0x190 [xfs]
> Jun 11 15:07:13 machine2 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 11 15:07:13 machine2 kernel:  xfs_log_force_seq+0x91/0x180 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  __x64_sys_fdatasync+0x52/0xa0
> Jun 11 15:07:13 machine2 kernel:  do_syscall_64+0x39/0x90
> Jun 11 15:07:13 machine2 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 11 15:07:13 machine2 kernel: RIP: 0033:0x7fceaeb0d20a
> Jun 11 15:07:13 machine2 kernel: RSP: 002b:00007fce93ffdb60 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
> Jun 11 15:07:13 machine2 kernel: RAX: ffffffffffffffda RBX: 0000559f4c37e6a8 RCX: 00007fceaeb0d20a
> Jun 11 15:07:13 machine2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007
> Jun 11 15:07:13 machine2 kernel: RBP: 0000559f4c37e618 R08: 0000000000000000 R09: 00000000000013bb
> Jun 11 15:07:13 machine2 kernel: R10: 0000559f4c3a51d0 R11: 0000000000000246 R12: 0000000000000008
> Jun 11 15:07:13 machine2 kernel: R13: 0000559f4c33a828 R14: 00007fce93ffdc20 R15: 0000000000000000
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: INFO: task nix:804252 blocked for more than 122 seconds.
> Jun 11 15:07:13 machine2 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 11 15:07:13 machine2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 11 15:07:13 machine2 kernel: task:nix             state:D stack:0     pid:804252 ppid:804251 flags:0x00000002
> Jun 11 15:07:13 machine2 kernel: Call Trace:
> Jun 11 15:07:13 machine2 kernel:  <TASK>
> Jun 11 15:07:13 machine2 kernel:  __schedule+0x3fc/0x1440
> Jun 11 15:07:13 machine2 kernel:  ? kmem_cache_alloc+0x19d/0x390
> Jun 11 15:07:13 machine2 kernel:  ? xlog_ticket_alloc+0x2d/0xb0 [xfs]
> Jun 11 15:07:13 machine2 kernel:  schedule+0x5e/0xe0
> Jun 11 15:07:13 machine2 kernel:  schedule_preempt_disabled+0x15/0x30
> Jun 11 15:07:13 machine2 kernel:  rwsem_down_write_slowpath+0x1f3/0x640
> Jun 11 15:07:13 machine2 kernel:  ? xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 11 15:07:13 machine2 kernel:  down_write+0x6b/0x70
> Jun 11 15:07:13 machine2 kernel:  xfs_trans_alloc_ichange+0x9d/0x200 [xfs]
> Jun 11 15:07:13 machine2 kernel:  xfs_setattr_nonsize+0x9a/0x390 [xfs]
> Jun 11 15:07:13 machine2 kernel:  notify_change+0x1f5/0x4c0
> Jun 11 15:07:13 machine2 kernel:  ? chown_common+0x22a/0x240
> Jun 11 15:07:13 machine2 kernel:  chown_common+0x22a/0x240
> Jun 11 15:07:13 machine2 kernel:  ksys_fchown+0x85/0xd0
> Jun 11 15:07:13 machine2 kernel:  __x64_sys_fchown+0x17/0x30
> Jun 11 15:07:13 machine2 kernel:  do_syscall_64+0x39/0x90
> Jun 11 15:07:13 machine2 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 11 15:07:13 machine2 kernel: RIP: 0033:0x7f3debd05ddb
> Jun 11 15:07:13 machine2 kernel: RSP: 002b:00007ffd20bf1c58 EFLAGS: 00000246 ORIG_RAX: 000000000000005d
> Jun 11 15:07:13 machine2 kernel: RAX: ffffffffffffffda RBX: 0000563449847d9d RCX: 00007f3debd05ddb
> Jun 11 15:07:13 machine2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007
> Jun 11 15:07:13 machine2 kernel: RBP: 0000000000080006 R08: 0000000000000001 R09: 0000000000000000
> Jun 11 15:07:13 machine2 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000007
> Jun 11 15:07:13 machine2 kernel: R13: 0000000000020042 R14: 000056344985f8e8 R15: 0000000000000000
> Jun 11 15:07:13 machine2 kernel:  </TASK>
> Jun 11 15:07:13 machine2 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
> Jun 11 15:07:13 machine2 systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
> Jun 11 15:07:13 machine2 systemd[1]: systemd-journald.service: Killing process 474 (systemd-journal) with signal SIGABRT.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Process 474 (systemd-journal) of user 0 terminated abnormally with signal 6/ABRT, processing...
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Process 474 (systemd-journal) of user 0 dumped core.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.be2bc38389c9490cb71c3494a7469dc7.474.1749647228000000.zst
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libzstd.so.1 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libaudit.so.1 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libattr.so.1 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libseccomp.so.2 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libpam.so.0 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libcrypt.so.2 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libcap.so.2 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Module libacl.so.1 without build-id.
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: Stack trace of thread 474:
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #0  0x00007f208f89513e __futex_abstimed_wait_common (libc.so.6 + 0x8d13e)
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #1  0x00007f208f89a5f3 __pthread_clockjoin_ex (libc.so.6 + 0x925f3)
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #2  0x00007f208fcd65ee journal_file_set_offline_thread_join (libsystemd-shared-256.so + 0x2d65ee)
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #3  0x00007f208fcd66d4 journal_file_set_online (libsystemd-shared-256.so + 0x2d66d4)
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #4  0x00007f208fcd95a7 journal_file_append_object (libsystemd-shared-256.so + 0x2d95a7)
> Jun 11 15:07:13 machine2 systemd-coredump[804693]: #5  0x00007f208fcdeb0f journal_file_append_data (libsystemd-shared-256.so + 0x2deb0f)

> Jun 12 14:41:26 machine3 kernel: INFO: task ib_io_wr-7:4619 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:ib_io_wr-7      state:D stack:0     pid:4619  ppid:1      flags:0x00004002
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_wait_on_iclog+0x174/0x190 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xfs_log_force_seq+0x91/0x180 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  __x64_sys_fsync+0x3b/0x70
> Jun 12 14:41:26 machine3 kernel:  do_syscall_64+0x39/0x90
> Jun 12 14:41:26 machine3 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 12 14:41:26 machine3 kernel: RIP: 0033:0x7f4dd8f0d6ea
> Jun 12 14:41:26 machine3 kernel: RSP: 002b:00007f4bb8ff62a0 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
> Jun 12 14:41:26 machine3 kernel: RAX: ffffffffffffffda RBX: 000000000000001d RCX: 00007f4dd8f0d6ea
> Jun 12 14:41:26 machine3 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000001d
> Jun 12 14:41:26 machine3 kernel: RBP: 00007f4bb8ff8530 R08: 0000000000000000 R09: 00000000000020ad
> Jun 12 14:41:26 machine3 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f4dc6641b70
> Jun 12 14:41:26 machine3 kernel: R13: 0000000000000000 R14: 00007f4dc6641f80 R15: 00000000046f7c08
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task ib_log_flush:4700 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:ib_log_flush    state:D stack:0     pid:4700  ppid:1      flags:0x00004002
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? __folio_batch_release+0x1f/0x60
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_wait_on_iclog+0x174/0x190 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xfs_log_force_seq+0x91/0x180 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  __x64_sys_fsync+0x3b/0x70
> Jun 12 14:41:26 machine3 kernel:  do_syscall_64+0x39/0x90
> Jun 12 14:41:26 machine3 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 12 14:41:26 machine3 kernel: RIP: 0033:0x7f4dd8f0d6ea
> Jun 12 14:41:26 machine3 kernel: RSP: 002b:00007f4b94b89690 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
> Jun 12 14:41:26 machine3 kernel: RAX: ffffffffffffffda RBX: 00007f4b94b8b930 RCX: 00007f4dd8f0d6ea
> Jun 12 14:41:26 machine3 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000018
> Jun 12 14:41:26 machine3 kernel: RBP: 00007f4b94b8b920 R08: 0000000000000000 R09: 0000000000000000
> Jun 12 14:41:26 machine3 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00000000046f4498
> Jun 12 14:41:26 machine3 kernel: R13: 0000000000000000 R14: 00007f4dc50be340 R15: 00000000046f7c08
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task ib_log_writer:4702 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:ib_log_writer   state:D stack:0     pid:4702  ppid:1      flags:0x00000002
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? kmem_cache_alloc+0x19d/0x390
> Jun 12 14:41:26 machine3 kernel:  ? xlog_ticket_alloc+0x2d/0xb0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  schedule_preempt_disabled+0x15/0x30
> Jun 12 14:41:26 machine3 kernel:  rwsem_down_write_slowpath+0x1f3/0x640
> Jun 12 14:41:26 machine3 kernel:  ? xfs_trans_alloc+0xe5/0x230 [xfs]
> Jun 12 14:41:26 machine3 kernel:  down_write+0x6b/0x70
> Jun 12 14:41:26 machine3 kernel:  xfs_vn_update_time+0xa1/0x1b0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  kiocb_modified+0x9b/0xd0
> Jun 12 14:41:26 machine3 kernel:  xfs_file_write_checks+0x1dd/0x280 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xfs_file_buffered_write+0x60/0x2b0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? selinux_file_permission+0x119/0x160
> Jun 12 14:41:26 machine3 kernel:  vfs_write+0x24f/0x430
> Jun 12 14:41:26 machine3 kernel:  __x64_sys_pwrite64+0xa4/0xd0
> Jun 12 14:41:26 machine3 kernel:  do_syscall_64+0x39/0x90
> Jun 12 14:41:26 machine3 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 12 14:41:26 machine3 kernel: RIP: 0033:0x7f4dd8eee02f
> Jun 12 14:41:26 machine3 kernel: RSP: 002b:00007f4b93b87280 EFLAGS: 00000246 ORIG_RAX: 0000000000000012
> Jun 12 14:41:26 machine3 kernel: RAX: ffffffffffffffda RBX: 00007f4b9b7f0c00 RCX: 00007f4dd8eee02f
> Jun 12 14:41:26 machine3 kernel: RDX: 0000000000000200 RSI: 00007f4b9b7f0c00 RDI: 0000000000000018
> Jun 12 14:41:26 machine3 kernel: RBP: 00007f4b93b895a0 R08: 0000000000000000 R09: 00007f4b93b895dc
> Jun 12 14:41:26 machine3 kernel: R10: 0000000000f71e00 R11: 0000000000000246 R12: 0000000000f71e00
> Jun 12 14:41:26 machine3 kernel: R13: 00007f4b9b7f0c00 R14: 00000000046f7be8 R15: 00007f4dc4307210
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task connection:87027 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:connection      state:D stack:0     pid:87027 ppid:1      flags:0x00004002
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? __folio_batch_release+0x1f/0x60
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_wait_on_iclog+0x174/0x190 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xfs_log_force_seq+0x91/0x180 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xfs_file_fsync+0x1a9/0x2b0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  __x64_sys_fdatasync+0x52/0xa0
> Jun 12 14:41:26 machine3 kernel:  do_syscall_64+0x39/0x90
> Jun 12 14:41:26 machine3 kernel:  entry_SYSCALL_64_after_hwframe+0x78/0xe2
> Jun 12 14:41:26 machine3 kernel: RIP: 0033:0x7f4dd8f0d20a
> Jun 12 14:41:26 machine3 kernel: RSP: 002b:00007f4b8a05b540 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
> Jun 12 14:41:26 machine3 kernel: RAX: ffffffffffffffda RBX: 00007f4b8a05b620 RCX: 00007f4dd8f0d20a
> Jun 12 14:41:26 machine3 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000003d
> Jun 12 14:41:26 machine3 kernel: RBP: 00007f4b8a05b610 R08: 0000000000000000 R09: 0000000000000000
> Jun 12 14:41:26 machine3 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000000000000003d
> Jun 12 14:41:26 machine3 kernel: R13: 0000000000000010 R14: 0000000000000000 R15: 00000000046e5f20
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u5:2:80717 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u5:2    state:D stack:0     pid:80717 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_order_write+0x134/0x170 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x5ab/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u5:3:81267 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u5:3    state:D stack:0     pid:81267 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? __schedule+0x404/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? select_task_rq_fair+0x6e5/0x1b30
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_order_write+0x134/0x170 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x5ab/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u6:3:83203 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u6:3    state:D stack:0     pid:83203 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  ? blk_mq_submit_bio+0x27b/0x660
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_state_get_iclog_space+0x107/0x2c0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_write_get_more_iclog_space+0xde/0x110 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xlog_write+0x34c/0x470 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x72c/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u5:0:85256 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u5:0    state:D stack:0     pid:85256 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_order_write+0x134/0x170 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x5ab/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u5:1:86241 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u5:1    state:D stack:0     pid:86241 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_order_write+0x134/0x170 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x5ab/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>
> Jun 12 14:41:26 machine3 kernel: INFO: task kworker/u6:4:86403 blocked for more than 122 seconds.
> Jun 12 14:41:26 machine3 kernel:       Not tainted 6.6.92 #1-NixOS
> Jun 12 14:41:26 machine3 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Jun 12 14:41:26 machine3 kernel: task:kworker/u6:4    state:D stack:0     pid:86403 ppid:2      flags:0x00004000
> Jun 12 14:41:26 machine3 kernel: Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
> Jun 12 14:41:26 machine3 kernel: Call Trace:
> Jun 12 14:41:26 machine3 kernel:  <TASK>
> Jun 12 14:41:26 machine3 kernel:  __schedule+0x3fc/0x1440
> Jun 12 14:41:26 machine3 kernel:  schedule+0x5e/0xe0
> Jun 12 14:41:26 machine3 kernel:  xlog_state_get_iclog_space+0x107/0x2c0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? load_balance+0x175/0xfd0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_default_wake_function+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  xlog_write+0x7a/0x470 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_xlog_cil_order_cmp+0x10/0x10 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? merge+0x66/0xc0
> Jun 12 14:41:26 machine3 kernel:  ? xlog_cil_order_write+0x9a/0x170 [xfs]
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_xlog_cil_order_cmp+0x10/0x10 [xfs]
> Jun 12 14:41:26 machine3 kernel:  xlog_cil_push_work+0x72c/0x8f0 [xfs]
> Jun 12 14:41:26 machine3 kernel:  process_one_work+0x18d/0x3a0
> Jun 12 14:41:26 machine3 kernel:  worker_thread+0x28c/0x3b0
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_worker_thread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  kthread+0xe8/0x120
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork+0x34/0x50
> Jun 12 14:41:26 machine3 kernel:  ? __pfx_kthread+0x10/0x10
> Jun 12 14:41:26 machine3 kernel:  ret_from_fork_asm+0x1b/0x30
> Jun 12 14:41:26 machine3 kernel:  </TASK>