Thanks for the info on removing stubborn dead OSDs. The actual syntax
required was:
cephadm rm-daemon --name osd.2 --fsid <fsid>--force
On the "too many pgs", that's because I'm down 2 OSDs. I've got new
drives, but they were waiting to clear out the dead stuff. I know it's
risky, but I have backups.
Recall that the start of this thread was on a HEALTH_OK system and
prometheus was not activating. The OSD stuff was just a distraction.
I did notice that the attempt to add a new mds did work after I did a
"ceph mgr fail", so it's only prometheus that's a permanent problem.
Here's the latest health after clearing out the dead OSDs:
# ceph health detail
HEALTH_ERR Module 'prometheus' has failed: gaierror(-2, 'Name or service
not known'); too many PGs per OSD (648 > max 560)
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2,
'Name or service not known')
Module 'prometheus' has failed: gaierror(-2, 'Name or service not
known')
[WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
And yes, disabling prometheus will make the "name or service not known"
errors go away.
On 3/28/25 02:49, Eugen Block wrote:
Did you disable the prometheus module? I would expect the warning to
clear if you did.
Somewhere deep inside ceph, those deleted OSDs still exist. Likely
because ceph08 hasn't deleted the systemd units that run them.
Or do you still see those OSDs in 'cephadm ls' output on ceph08? If
you do, and if those OSDs are really already drained/purged, you can
remove them with 'cephadm rm-daemon --name osd.2'. And I would try to
get the MGR into a working state first, before you try to deploy
prometheus again. So my recommendation is to get into HEALTH_OK first.
And btw, "TOO_MANY_PGS: too many PGs per OSD (648 > max 560)" is
serious, you can end up with inactive PGs during recovery, so I'd also
consider checking the pools and their PGs.
Zitat von Tim Holloway <timh@xxxxxxxxxxxxx>:
Thanks for your patience.
host ceph06 isn't referenced in the config database. I think I've
finally purged it. I also reset the dashboard API host address from
ceph08 to dell02. But since prometheus isn't running on dell02
either, there's no gain there.
I did clear some of that lint out via "ceph mgr fail".
So here's the latest. There are strange things happening at the base
OS level that keep host ceph08 from running its OSDs anymore. At
boot, device /dev/sdb suddenly changes to /dev/sdd (????) and there
seem to be I/O errors. It's really strange, but I'm going to replace
the physical drive and that will hopefully cure that.
The problem is, reef and earlier releases seem to have a lot of
trouble in deleting OSDs that aren't running. As I've noted before,
they tend to get permanently stuck in the "deleting" state. When I
cannot restart the OSD, the only cure for that has been to run around
the system and apply brute force until things clear up.
I did a dashboard purge of the OSDs on ceph08 and that removed them
from the GUI (they'd already drained). I also banged on things until
I got them out of the OSD tree display and then did a crush delete on
host ceph08. And, incidentally, the OSD tree works on simple host
names, not FQDNs like the rest of ceph!
So in theory, I'm ready to jack in new drives and add new OSDs to
ceph08. Except:
# ceph health detail
HEALTH_ERR 2 failed cephadm daemon(s); Module 'prometheus' has
failed: gaierror(-2, 'Name or service not known'); too many PGs per
OSD (648 > max 560)
[WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)
daemon osd.2 on ceph08.internal.mousetech.com is in error state
daemon osd.4 on ceph08.internal.mousetech.com is in error state
[ERR] MGR_MODULE_ERROR: Module 'prometheus' has failed: gaierror(-2,
'Name or service not known')
Module 'prometheus' has failed: gaierror(-2, 'Name or service not
known')
[WRN] TOO_MANY_PGS: too many PGs per OSD (648 > max 560)
Somewhere deep inside ceph, those deleted OSDs still exist. Likely
because ceph08 hasn't deleted the systemd units that run them.
I'm going to try removing/re-installing prometheus. since it's now
showing up in ceph health. I think last time I had zombie OSDs I had
to brute-force delete their corresponding directories under
/var/lib/ceph.
On 3/27/25 14:01, Eugen Block wrote:
ceph config-key rm
mgr/cephadm/host.ceph06.internal.mousetech.com.devices.0
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
