Hi Mikael,
Thanks for the report. I was also considering upgrading from 19.2.2 to
19.23. Should be related to a change between those 2 versions as I
experienced no problem during the 18.2.7 to 19.2.2. upgrade... It
reminds me a problem in one of the Quincy update if I'm right with
something similar (but probably a different cause) where the device
activation was doing just too many times the same command (at that time
was a trivial indentation issue in the code)... but at least it seems
that activation of many OSDs per node was unsufficiently tested. I don't
know if testing was improved...
Best regards,
Michel
Le 14/09/2025 à 10:23, Eugen Block a écrit :
This is interesting, I was planning to upgrade our own cluster next
week from 18.2.7 to 19.2.3 as well, now I'm hesitating. Although we
don't have that many OSDs per node, so we probably will not have this
issue you're describing. But I can confirm that 'cephadm ceph-volume
raw list' on my virtual test environment with only 3 OSDs per node
takes around 11 seconds (and empty output). On Reef the output is not
empty (probably because exclude_lvm_osd_devices is not present there
as I understand it) and it only takes 4 seconds to complete with
around 10 OSDs per node.
I'll have to check with my colleagues if we should still move forward
with the upgrade...
Thanks for reporting that! Did you check if there's a tracker issue
for that?
Thanks,
Eugen
Zitat von Mikael Öhman <micketeer@xxxxxxxxx>:
I'm fighting with a ceph upgrade, going 18.2.7 to 19.2.3.
This time again the ceph-volume activate step is taking to long,
triggering
failures due to systemd service timing out so the orch daemon fails
(though
the osd does eventually come up, the daemon is still dead, and upgrade
halts).
I can also reproduce the slowdown of startup with
cephadm ceph-volume raw list
(I don't use raw devices, but the ceph-volume activation method
hardcodes
checking raw first
https://github.com/ceph/ceph/blob/4d5ad8c1ef04f38d14402f0d89f2df2b7d254c2c/src/ceph-volume/ceph_volume/activate/main.py#L46
)
That's takes 6s on 18.2.7, but 4m32s minutes on 19.2.3 !
I have 42 spinning drives per host (with multipath).
It's spending all of it's time in the new method:
self.exclude_lvm_osd_devices()
and the list of items to scan, given all the duplication from
multipath +
and mapper names, it ends up with 308 items to scan in my setup.
With good old print debugging, i found that while the threadpool speeds
things up a bit, it simply takes to long to construct all those Device()
objects.
In fact, just creating a single Device() object, since it needs to call
disk.get_devices()
at least once, since this list does not include all devices, it
filters out
things like
"/dev/mapper/mpathxx" from the list, but the code always regenerates
(the
same) device list if the path isn't found:
if not sys_info.devices.get(self.path):
sys_info.devices = disk.get_devices()
will now force it to re-generate this list >400 times (initial 32
times in
parallel, followed by about 400 more which will never match the device
name).
In the end, it's again O(n^2) computational time to list these raw
devices
with ceph-volume.
So with 32 threads in the pool, it's also now requires running heavy
load
for 5 minutes before completing this trivial task every time the deamon
needs to start.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
