Re: bdev_ioring -- true - Drives failing

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks for your reply… 

Currently I disabled that option,, but below are the logs…. 

# zcat ceph-osd.*.log.1.gz |egrep -i "faile|error|down"
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:27.303+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:27.303+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.307+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2025-07-11T17:57:27.307+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.311+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:27.311+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.311+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:27.311+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.315+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:27.315+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.319+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:27.319+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.323+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:27.323+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.327+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:27.327+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.331+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:27.331+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.335+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:27.335+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.343+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:27.347+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:27.355+0000 7f2aeacb4680  1 freelist shutdown
2025-07-11T17:57:27.355+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:27.355+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:27.995+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:28.083+0000 7f2aeacb4680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:57:28.103+0000 7f2aeacb4680  0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:57:28.487+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:28.487+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:28.783+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:29.559+0000 7f2aeacb4680  1 bluestore(/var/lib/ceph/osd/ceph-4/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.670384 seconds
2025-07-11T17:57:29.563+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:29.563+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:29.735+0000 7f2aeacb4680  1 freelist shutdown
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:30.155+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:30.227+0000 7f2aeacb4680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:57:30.227+0000 7f2aeacb4680  0 bluestore(/var/lib/ceph/osd/ceph-4/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:57:30.379+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:30.379+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:30.687+0000 7f2aeacb4680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:30.695+0000 7f2aeacb4680  3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory
2025-07-11T17:57:30.755+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:30.755+0000 7f2aeacb4680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:31.639+0000 7f2aeacb4680  1 freelist shutdown
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:44.583+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:44.583+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2025-07-11T17:57:44.587+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:44.587+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.591+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:44.591+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.591+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:44.591+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.595+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:44.595+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.599+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:44.599+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.607+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:44.607+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.611+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:44.611+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.615+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:44.615+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.623+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:44.623+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:44.627+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:44.635+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:44.647+0000 7f214047d680  1 freelist shutdown
2025-07-11T17:57:44.647+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:44.647+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.183+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:45.183+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.187+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:45.187+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:45.267+0000 7f214047d680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:57:45.291+0000 7f214047d680  0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:57:45.675+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:45.675+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:45.991+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:45.991+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:46.703+0000 7f214047d680  1 bluestore(/var/lib/ceph/osd/ceph-5/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.604180 seconds
2025-07-11T17:57:46.703+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:46.703+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:46.911+0000 7f214047d680  1 freelist shutdown
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:47.327+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:47.327+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:47.399+0000 7f214047d680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:57:47.399+0000 7f214047d680  0 bluestore(/var/lib/ceph/osd/ceph-5/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:57:47.579+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:47.579+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.863+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:57:47.863+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.867+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:57:47.867+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.867+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:57:47.867+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.867+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:57:47.867+0000 7f214047d680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:57:47.867+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:57:47.867+0000 7f214047d680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:57:47.871+0000 7f214047d680  3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory
2025-07-11T17:57:47.931+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:57:47.931+0000 7f214047d680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:57:48.731+0000 7f214047d680  1 freelist shutdown
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:03.399+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:03.399+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.403+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:03.403+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2025-07-11T17:58:03.403+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:03.403+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.407+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:03.407+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.407+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:03.407+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.411+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:03.411+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.415+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:03.415+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.419+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:03.419+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.423+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:03.423+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.427+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:03.427+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:03.431+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:03.435+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:03.439+0000 7f082a781680  1 freelist shutdown
2025-07-11T17:58:03.439+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:03.439+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:04.055+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:04.055+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:04.127+0000 7f082a781680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:58:04.147+0000 7f082a781680  0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:58:04.459+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:04.459+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:04.583+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:04.583+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:05.243+0000 7f082a781680  1 bluestore(/var/lib/ceph/osd/ceph-6/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.553290 seconds
2025-07-11T17:58:05.243+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:05.243+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:05.295+0000 7f082a781680  1 freelist shutdown
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:05.711+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:05.711+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:05.783+0000 7f082a781680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:58:05.783+0000 7f082a781680  0 bluestore(/var/lib/ceph/osd/ceph-6/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:58:05.935+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:05.935+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:06.247+0000 7f082a781680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:06.247+0000 7f082a781680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:06.255+0000 7f082a781680  3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory
2025-07-11T17:58:06.315+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:06.315+0000 7f082a781680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:07.251+0000 7f082a781680  1 freelist shutdown
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:26.279+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:26.279+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.283+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
2025-07-11T17:58:26.283+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.283+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:26.283+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.287+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:26.287+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.287+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:26.287+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.287+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:26.291+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.291+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:26.291+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.295+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:26.295+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.295+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:26.299+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.299+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:26.299+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.303+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:26.307+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:26.311+0000 7f058d726680  1 freelist shutdown
2025-07-11T17:58:26.311+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:26.311+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:26.943+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:26.943+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:27.023+0000 7f058d726680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:58:27.039+0000 7f058d726680  0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:58:27.347+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:27.347+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:27.455+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:27.455+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:28.127+0000 7f058d726680  1 bluestore(/var/lib/ceph/osd/ceph-7/) _fsck_on_open <<<FINISH>>> with 0 errors, 0 warnings, 0 repaired, 0 remaining in 0.565725 seconds
2025-07-11T17:58:28.127+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:28.131+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:28.179+0000 7f058d726680  1 freelist shutdown
2025-07-11T17:58:28.579+0000 7f058d726680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:28.579+0000 7f058d726680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:28.579+0000 7f058d726680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:28.579+0000 7f058d726680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:28.579+0000 7f058d726680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:28.583+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:28.583+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:28.655+0000 7f058d726680  1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2
2025-07-11T17:58:28.655+0000 7f058d726680  0 bluestore(/var/lib/ceph/osd/ceph-7/) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) ...
2025-07-11T17:58:28.795+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:28.795+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:                         Options.error_if_exists: 0
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:                Options.max_bgerror_resume_count: 2147483647
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:            Options.bgerror_resume_retry_interval: 1000000
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:             Options.allow_data_in_errors: 0
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2025-07-11T17:58:29.103+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default)
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.103+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0)
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.103+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1)
2025-07-11T17:58:29.103+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.103+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2)
2025-07-11T17:58:29.107+0000 7f058d726680  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L)
2025-07-11T17:58:29.107+0000 7f058d726680  2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P)
2025-07-11T17:58:29.111+0000 7f058d726680  3 rocksdb: [db/db_impl/db_impl_open.cc:1432] Failed to truncate log #4: IO error: No such file or directory: While open a file for appending: db.wal/000004.log: No such file or directory
2025-07-11T17:58:29.171+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:496] Shutdown: canceling all background work
2025-07-11T17:58:29.171+0000 7f058d726680  4 rocksdb: [db/db_impl/db_impl.cc:704] Shutdown complete
2025-07-11T17:58:30.015+0000 7f058d726680  1 freelist shutdown

Regards
Dev

> On Jul 24, 2025, at 10:57 AM, Joshua Blanch <joshua.blanch@xxxxxxxxx> wrote:
> 
> r with that config value but I’ve seen a bug in that version that causes osds t

_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux