No. Same issue unfortunately.
On Saturday, Apr 05, 2025 at 1:27 PM, Eugen Block <eblock@xxxxxx> wrote:I don't see the module error in the logs you provided. Has it cleared?
A mgr failover is often helpful in such cases, basically it's the
first thing I suggest since two or three years.
Zitat von Jeremy Hansen <jeremy@xxxxxxxxxx>:
Thank you so much for the detailed instructions. Here’s logs from
the failover to a new node.
Apr 05 20:06:08 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:08 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:08.568+0000 7efcbd800700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.72 - - [05/Apr/2025:20:06:08] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:10 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:10 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:10.702+0000 7efcb4c00700 0 [prometheus INFO cherrypy.access.139625005087208] ::ffff:192.168.47.71 - - [05/Apr/2025:20:06:10] "GET /metrics HTTP/1.1" 200 - ""
"Prometheus/2.33.4"
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.027+0000 7efd95200700 1 mgr handle_mgr_map
Activating!
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.028+0000 7efd95200700 1 mgr handle_mgr_map I am now
activating
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] Stopping
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.040+0000 7efc27e00700 0 [dashboard INFO root] Stopped
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] Stopping
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc27e00700 0 [prometheus INFO root] Stopped
engine
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STOPPING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.041+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STOPPING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.053+0000 7efc3a400700 0 [dashboard INFO root] Engine
stopped.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut
down
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut
down
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STOPPED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STOPPED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.054+0000 7efc43000700 0 [prometheus INFO root] Engine
stopped.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.056+0000 7efc27e00700 0 [alerts DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 0 [alerts INFO root]
Init
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.057+0000 7efc27e00700 1 mgr load Constructed class from module:
alerts
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 0 [balancer DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efc27e00700 1 mgr load Constructed class from module:
balancer
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.058+0000 7efcfac00700 0 [alerts INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efc27e00700 0 [cephadm DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.060+0000 7efcb4c00700 0 [balancer INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Optimize plan
auto_2025-04-05_20:06:12
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Mode upmap, max misplaced
0.050000
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.067+0000 7efcb4c00700 0 [balancer INFO root] Some PGs (1.000000) are unknown; try again
later
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.212+0000 7efc27e00700 1 mgr load Constructed class from module:
cephadm
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 0 [crash DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.213+0000 7efc27e00700 1 mgr load Constructed class from module:
crash
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 0 [dashboard DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.215+0000 7efc27e00700 1 mgr load Constructed class from module:
dashboard
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.216+0000 7efc23800700 0 [dashboard INFO access_control] Loading user roles DB
version=2
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO sso] Loading SSO DB
version=1
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.217+0000 7efc23800700 0 [dashboard INFO root] server: ssl=yes host=::
port=8443
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 0 [devicehealth DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.218+0000 7efc27e00700 1 mgr load Constructed class from module:
devicehealth
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 0 [insights DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc27e00700 1 mgr load Constructed class from module:
insights
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc20600700 0 [insights INFO root] Reset curr health slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:12 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.219+0000 7efc21a00700 0 [devicehealth INFO root]
Starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 0 [iostat DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.222+0000 7efc27e00700 1 mgr load Constructed class from module:
iostat
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.224+0000 7efc27e00700 0 [mirroring DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.227+0000 7efc21a00700 0 [devicehealth INFO root] Check
health
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.235+0000 7efc27e00700 1 mgr load Constructed class from module:
mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 0 [nfs DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.236+0000 7efc27e00700 1 mgr load Constructed class from module:
nfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 0 [orchestrator DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.237+0000 7efc27e00700 1 mgr load Constructed class from module:
orchestrator
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 0 [pg_autoscaler DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc27e00700 1 mgr load Constructed class from module:
pg_autoscaler
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.239+0000 7efc10200700 0 [pg_autoscaler INFO root]
_maybe_adjust
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 0 [progress DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.240+0000 7efc27e00700 1 mgr load Constructed class from module:
progress
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.247+0000 7efc27e00700 0 [prometheus DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.248+0000 7efc27e00700 1 mgr load Constructed class from module:
prometheus
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc0ee00700 0 [progress INFO root]
Loading...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.252+0000 7efc27e00700 0 [rbd_support DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded [<progress.module.GhostEvent object at 0x7efcf2127710>, <progress.module.GhostEvent object at 0x7efcf2127cc0>, <progress.module.GhostEvent object at 0x7efcf2127208>, <progress.module.GhostEvent object at 0x7efcf2127f28>, <progress.module.GhostEvent object at 0x7efcf21270f0>, <progress.module.GhostEvent object at 0x7efcf2127eb8>, <progress.module.GhostEvent object at 0x7efcf2127b38>, <progress.module.GhostEvent object at 0x7efcf2127a20>, <progress.module.GhostEvent object at 0x7efcf2127da0>, <progress.module.GhostEvent object at 0x7efcf2127f98>, <progress.module.GhostEvent object at 0x7efcf2127780>, <progress.module.GhostEvent object at 0x7efcf2127320>, <progress.module.GhostEvent object at 0x7efcf2127588>, <progress.module.GhostEvent object at 0x7efcf2127518>, <progress.module.GhostEvent object at 0x7efcf2127c50>, <progress.module.GhostEvent object at 0x7efcf21279e8>, <progress.module.GhostEvent object at 0x7efcf2127d68>, <progress.module.GhostEvent object at 0x7efcf2127470>, <progress.module.GhostEvent object at 0x7efcf2127940>, <progress.module.GhostEvent object at 0x7efcf2127048>, <progress.module.GhostEvent object at 0x7efcf2127390>, <progress.module.GhostEvent object at 0x7efcf21276d8>, <progress.module.GhostEvent object at 0x7efcf2127a90>, <progress.module.GhostEvent object at 0x7efcf2127ba8>, <progress.module.GhostEvent object at 0x7efcf2127e48>, <progress.module.GhostEvent object at 0x7efcf21272e8>, <progress.module.GhostEvent object at 0x7efcf2127080>, <progress.module.GhostEvent object at 0x7efcf2127b00>, <progress.module.GhostEvent object at 0x7efcf21279b0>, <progress.module.GhostEvent object at 0x7efcf20c1588>, <progress.module.GhostEvent object at 0x7efcf20c1ac8>, <progress.module.GhostEvent object at 0x7efcf20c1550>, <progress.module.GhostEvent object at 0x7efcf20c17f0>, <progress.module.GhostEvent object at 0x7efcf20c1d68>, <progress.module.GhostEvent object at 0x7efcf20c1b70>, <progress.module.GhostEvent object at 0x7efcf20c1c50>, <progress.module.GhostEvent object at 0x7efcf20c1860>, <progress.module.GhostEvent object at 0x7efcf20c1908>, <progress.module.GhostEvent object at 0x7efcf20c14e0>, <progress.module.GhostEvent object at 0x7efcf20c1cf8>, <progress.module.GhostEvent object at 0x7efcf20c1470>, <progress.module.GhostEvent object at 0x7efcf20c1d30>, <progress.module.GhostEvent object at 0x7efcf20c1e80>, <progress.module.GhostEvent object at 0x7efcf20c1e10>, <progress.module.GhostEvent object at 0x7efcf20c1eb8>, <progress.module.GhostEvent object at 0x7efcf20c1f28>, <progress.module.GhostEvent object at 0x7efcf20c1668>, <progress.module.GhostEvent object at 0x7efcf20c16a0>, <progress.module.GhostEvent object at 0x7efcf20c1ba8>, <progress.module.GhostEvent object at 0x7efcf20c1c18>] historic
events
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Loaded OSDMap,
ready.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.253+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap change
27179..27179
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] server_addr: :: server_port:
9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc0d600700 0 [prometheus INFO root] Cache
enabled
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.256+0000 7efc08600700 0 [prometheus INFO root] starting metric collection
thread
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO root] Starting
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STARTING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.258+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STARTING
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] recovery thread
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc07200700 0 [rbd_support INFO root] starting
setup
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.259+0000 7efc27e00700 1 mgr load Constructed class from module:
rbd_support
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 0 [restful DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.301+0000 7efc27e00700 1 mgr load Constructed class from module:
restful
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.303+0000 7efc07200700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler:
load_schedules
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.305+0000 7efc07200700 0 [rbd_support INFO root] load_schedules: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.307+0000 7efc04a00700 0 [restful INFO root] server_addr: :: server_port:
8003
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.310+0000 7efc27e00700 0 [snap_schedule DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.312+0000 7efc04a00700 0 [restful WARNING root] server not running: no certificate
configured
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.73' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.319+0000 7efc23800700 0 [dashboard INFO root] Configured CherryPy, starting
engine...
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.72' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: Warning: Permanently added '192.168.47.71' (ECDSA) to the list of known
hosts.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.361+0000 7efbf8200700 0 [rbd_support INFO root] MirrorSnapshotScheduleHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.369+0000 7efbf7800700 0 [rbd_support INFO root] PerfHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.371+0000 7efc07200700 0 [rbd_support INFO root] load_task_task: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.389+0000 7efc27e00700 1 mgr load Constructed class from module:
snap_schedule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 0 [status DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.390+0000 7efc27e00700 1 mgr load Constructed class from module:
status
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.391+0000 7efc27e00700 0 [telemetry DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.392+0000 7efc27e00700 1 mgr load Constructed class from module:
telemetry
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.393+0000 7efc27e00700 0 [volumes DEBUG root] setting log level based on debug_mgr: INFO
(2/5)
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.409+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume
'cssnfsxyz'
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 0 [volumes INFO volumes.fs.async_job] queuing job for volume
'cssnfsxyz'
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.410+0000 7efc27e00700 1 mgr load Constructed class from module:
volumes
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: CherryPy
Checker:
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: The Application mounted at '' has an empty
config.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]:
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Serving on
http://:::9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Serving on
http://:::9283
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: [05/Apr/2025:20:06:12] ENGINE Bus
STARTED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO cherrypy.error] [05/Apr/2025:20:06:12] ENGINE Bus
STARTED
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.413+0000 7efc0d600700 0 [prometheus INFO root] Engine
started.
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.416+0000 7efbec400700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.417+0000 7efbe9200700 -1 client.0 error registering admin socket command: (17) File
exists
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.431+0000 7efbd8400700 0 [rbd_support INFO root] TaskHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.432+0000 7efc07200700 0 [rbd_support INFO root] TrashPurgeScheduleHandler:
load_schedules
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.434+0000 7efc07200700 0 [rbd_support INFO root] load_schedules: csxyz,
start_after=
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.445+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFS ->
/api/cephfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.448+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CephFsUi ->
/ui-api/cephfs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiUi ->
/ui-api/iscsi
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Iscsi ->
/api/iscsi
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.450+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: IscsiTarget ->
/api/iscsi/target
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.451+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaCluster ->
/api/nfs-ganesha/cluster
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.452+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaExports ->
/api/nfs-ganesha/export
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: NFSGaneshaUi ->
/ui-api/nfs-ganesha
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.453+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rbd ->
/api/block/image
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdStatus ->
/ui-api/block/rbd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.455+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdSnapshot ->
/api/block/image/{image_spec}/snap
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.456+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdTrash ->
/api/block/image/trash
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.457+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdNamespace ->
/api/block/pool/{pool_name}/namespace
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.458+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Pool ->
/api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.460+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PoolUi ->
/ui-api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.461+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RBDPool ->
/api/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Orchestrator ->
/ui-api/orchestrator
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.462+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Service ->
/api/service
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroring ->
/api/block/mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.463+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringSummary ->
/api/block/mirroring/summary
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolMode ->
/api/block/mirroring/pool
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolBootstrap ->
/api/block/mirroring/pool/{pool_name}/bootstrap
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.464+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringPoolPeer ->
/api/block/mirroring/pool/{pool_name}/peer
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.465+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirroringStatus ->
/ui-api/block/mirroring
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Rgw ->
/ui-api/rgw
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwDaemon ->
/api/rgw/daemon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.466+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwSite ->
/api/rgw/site
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.467+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwBucket ->
/api/rgw/bucket
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.468+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwUser ->
/api/rgw/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Auth ->
/api/auth
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.472+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Cluster ->
/api/cluster
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.473+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ClusterConfiguration ->
/api/cluster_conf
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.474+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRule ->
/api/crush_rule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: CrushRuleUi ->
/ui-api/crush_rule
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.476+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Daemon ->
/api/daemon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Docs ->
/docs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.477+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfile ->
/api/erasure_code_profile
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: ErasureCodeProfileUi ->
/ui-api/erasure_code_profile
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FrontendLogging ->
/ui-api/logging
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.478+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Grafana ->
/api/grafana
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.479+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Host ->
/api/host
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.480+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HostUi ->
/ui-api/host
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Health ->
/api/health
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: HomeController ->
/
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.481+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LangsController ->
/ui-api/langs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: LoginController ->
/ui-api/login
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Logs ->
/api/logs
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.482+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrModules ->
/api/mgr/module
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.483+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Monitor ->
/api/monitor
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.484+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Osd ->
/api/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.486+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdUi ->
/ui-api/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.487+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdFlagsController ->
/api/osd/flags
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MdsPerfCounter ->
/api/perf_counters/mds
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.488+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MonPerfCounter ->
/api/perf_counters/mon
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: OsdPerfCounter ->
/api/perf_counters/osd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.489+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RgwPerfCounter ->
/api/perf_counters/rgw
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: RbdMirrorPerfCounter ->
/api/perf_counters/rbd-mirror
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.490+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MgrPerfCounter ->
/api/perf_counters/mgr
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: TcmuRunnerPerfCounter ->
/api/perf_counters/tcmu-runner
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PerfCounters ->
/api/perf_counters
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.491+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusReceiver ->
/api/prometheus_receiver
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.492+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Prometheus ->
/api/prometheus
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: PrometheusNotifications ->
/api/prometheus/notifications
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.493+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Role ->
/api/role
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.494+0000 7efbd7a00700 0 [rbd_support INFO root] TrashPurgeScheduleHandler:
starting
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc07200700 0 [rbd_support INFO root] setup
complete
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.495+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Scope ->
/ui-api/scope
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Saml2 ->
/auth/saml2
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.496+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Settings ->
/api/settings
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.497+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: StandardSettings ->
/ui-api/standard_settings
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Summary ->
/api/summary
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Task ->
/api/task
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.498+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: Telemetry ->
/api/telemetry
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.499+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: User ->
/api/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserPasswordPolicy ->
/api/user
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.500+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: UserChangePassword ->
/api/user/{username}
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: FeatureTogglesEndpoint ->
/api/feature_toggles
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.501+0000 7efc23800700 0 [dashboard INFO dashboard.controllers._base_controller] Initializing controller: MessageOfTheDay ->
/ui-api/motd
Apr 05 20:06:12 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:12.710+0000 7efc23800700 0 [dashboard INFO dashboard.module] Engine
started.
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.039+0000 7efc27400700 0 log_channel(cluster) log [DBG] : pgmap v3: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714 ""
"Prometheus/2.33.4"
Apr 05 20:06:13 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:13.287+0000 7efc02200700 0 [prometheus INFO cherrypy.access.139624858224400] ::ffff:192.168.47.73 - - [05/Apr/2025:20:06:13] "GET /metrics HTTP/1.1" 200 39714 ""
"Prometheus/2.33.4"
Apr 05 20:06:14 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:14.037+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v4: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:16 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:16.038+0000 7efc26000700 0 log_channel(cluster) log [DBG] : pgmap v5: 609 pgs: 609 active+clean; 1.1 TiB data, 3.3 TiB used, 106 TiB / 109 TiB
avail
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.106+0000 7efc1fc00700 0 [insights INFO root] Received health check update 0
pending
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Applying 1 health updates to slot key health_history/2025-04-05_20 next flush None checks check count Apr 05 20:06:17 cn02.ceph.xyz.corp ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.149+0000 7efc20600700 0 [insights INFO root] Maybe flushing slot key health_history/2025-04-05_20 next flush None checks check count 0 needed
False
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.256+0000 7efc0ee00700 0 [progress INFO root] Processing OSDMap change
27179..27179
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 [cephadm INFO root] Detected new or changed devices on
cn01.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.518+0000 7efce9400700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn01.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 [cephadm INFO root] Detected new or changed devices on
cn03.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.611+0000 7efcd7c00700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn03.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 [cephadm INFO root] Detected new or changed devices on
cn02.ceph.xyz.corp
Apr 05 20:06:17 cn02.ceph.xyz.corp
ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn02-ceph-xyz-corp-ggixgj[2357414]: debug 2025-04-05T20:06:17.726+0000 7efccf000700 0 log_channel(cephadm) log [INF] : Detected new or changed devices on
cn02.ceph.xyz.corp
On Saturday, Apr 05, 2025 at 4:51 AM, Eugen Block <eblock@xxxxxx
(mailto:eblock@xxxxxx)> wrote:
Please don't drop the list from your response.
The orchestrator might be down, but you should have an active mgr
(ceph -s). On the host running the mgr, you can just run 'cephadm logs
--name mgr.<your_mgr>', or 'journalctl -u
ceph-<ceph_fsid>@mgr.<your_mgr>'.
To get fresh logs, you can just run 'ceph mgr fail', look on which
host a different mgr takes over and gather logs from there.
Zitat von Jeremy Hansen <jeremy@xxxxxxxxxx>:
Thanks for the clarification on version jumps. I interpreted the
docs to mean that I had to incrementally step through each release.
This will save a lot of time.
Since the orchestrator is down and I’m not sure how to trigger a
failure, I don’t know how to gather those logs. I apologize. I
appreciate your assistance.
-jeremy
On Saturday, Apr 05, 2025 at 4:01 AM, Eugen Block <eblock@xxxxxx
(mailto:eblock@xxxxxx)> wrote:
It might be a different issue, can you paste the entire stack trace
from the mgr when it's failing?
Also, you could go directly from Pacific to Reef, there's no need to
upgrade to Quincy (which is EOL). And I would also recommend to
upgrade to a latest major version, e.g. 17.2.8, not .0, otherwise you
might hit bugs which have already been fixed in later releases.
Zitat von Jeremy Hansen <jeremy@xxxxxxxxxx>:
I ran in to the “Error ENOENT: Module not found” issue with the
orchestrator. I see the note in the cephadm upgrade docs but I don’t
quite know what action to take to fix this:
ceph versions
{
"mon": {
"ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
pacific (stable)": 3
},
"mgr": {
"ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
pacific (stable)": 1,
"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
quincy (stable)": 2
},
"osd": {
"ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
pacific (stable)": 15
},
"mds": {
"ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
pacific (stable)": 3
},
"overall": {
"ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516)
pacific (stable)": 22,
"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e)
quincy (stable)": 2
}
}
ceph orch upgrade status
Error ENOENT: Module not found
ceph config-key get mgr/cephadm/osd_remove_queue
[]
Thanks. I’m trying to upgrade this cluster all the way to current.
-jeremy
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
