So right now I have this: pve01:~# find mon-store/ mon-store/ mon-store/ceph-0 mon-store/ceph-0/kv_backend mon-store/ceph-0/store.db mon-store/ceph-0/store.db/LOCK mon-store/ceph-0/store.db/IDENTITY mon-store/ceph-0/store.db/CURRENT mon-store/ceph-0/store.db/000004.log mon-store/ceph-0/store.db/MANIFEST-000005 mon-store/ceph-0/store.db/OPTIONS-000007 mon-store/ceph-1 mon-store/ceph-1/kv_backend mon-store/ceph-1/store.db mon-store/ceph-1/store.db/LOCK mon-store/ceph-1/store.db/IDENTITY mon-store/ceph-1/store.db/CURRENT mon-store/ceph-1/store.db/000004.log mon-store/ceph-1/store.db/MANIFEST-000005 mon-store/ceph-1/store.db/OPTIONS-000007 pve02:~# find mon-store/ mon-store/ mon-store/ceph-2 mon-store/ceph-2/kv_backend mon-store/ceph-2/store.db mon-store/ceph-2/store.db/LOCK mon-store/ceph-2/store.db/IDENTITY mon-store/ceph-2/store.db/CURRENT mon-store/ceph-2/store.db/000004.log mon-store/ceph-2/store.db/MANIFEST-000005 mon-store/ceph-2/store.db/OPTIONS-000007 mon-store/ceph-3 mon-store/ceph-3/kv_backend mon-store/ceph-3/store.db mon-store/ceph-3/store.db/LOCK mon-store/ceph-3/store.db/IDENTITY mon-store/ceph-3/store.db/CURRENT mon-store/ceph-3/store.db/000004.log mon-store/ceph-3/store.db/MANIFEST-000005 mon-store/ceph-3/store.db/OPTIONS-000007 pve03:~# find mon-store/ mon-store/ mon-store/ceph-4 mon-store/ceph-4/kv_backend mon-store/ceph-4/store.db mon-store/ceph-4/store.db/LOCK mon-store/ceph-4/store.db/IDENTITY mon-store/ceph-4/store.db/CURRENT mon-store/ceph-4/store.db/000004.log mon-store/ceph-4/store.db/MANIFEST-000005 mon-store/ceph-4/store.db/OPTIONS-000007 mon-store/ceph-5 mon-store/ceph-5/kv_backend mon-store/ceph-5/store.db mon-store/ceph-5/store.db/LOCK mon-store/ceph-5/store.db/IDENTITY mon-store/ceph-5/store.db/CURRENT mon-store/ceph-5/store.db/000004.log mon-store/ceph-5/store.db/MANIFEST-000005 mon-store/ceph-5/store.db/OPTIONS-000007 --- Em qua., 20 de ago. de 2025 às 15:15, Gilberto Ferreira < gilberto.nunes32@xxxxxxxxx> escreveu: > Ok... > I am doing it again. > I have 2 osd per node. > Do I need to create multiple folder for each osd? > Like > node1: > mon-store/ceph-osd0 > mon-store/ceph-osd1 > node2: > mon-store/ceph-osd2 > mon-store/ceph-osd3 > node3: > mon-store/ceph-osd3 > mon-store/ceph-osd3 > > And than rsynced everything to one node let's say: > nove1: /root/mon-store? > > Which one I should use in order to restore or recreate the mon? > > Sorry for so many questions. > I am trying to understand the whole process, so bare with me. > > Thanks for your patience. > > > > --- > > > Gilberto Nunes Ferreira > +55 (47) 99676-7530 - Whatsapp / Telegram > > > > > > > Em qua., 20 de ago. de 2025 às 14:35, Eugen Block <eblock@xxxxxx> > escreveu: > >> I feel like there's still a misunderstanding here. >> >> The mentioned procedure is: >> >> ms=/root/mon-store >> mkdir $ms >> >> # collect the cluster map from stopped OSDs >> for host in $hosts; do >> rsync -avz $ms/. user@$host:$ms.remote >> rm -rf $ms >> ssh user@$host <<EOF >> for osd in /var/lib/ceph/osd/ceph-*; do >> ceph-objectstore-tool --data-path \$osd --no-mon-config --op >> update-mon-db --mon-store-path $ms.remote >> done >> EOF >> rsync -avz user@$host:$ms.remote/. $ms >> done >> >> >> It collects the clustermap on each host, querying each OSD, but then >> it "merges" it into one store, the local $ms store. That is used then >> to start up the first monitor. So however you do this, make sure you >> have all the clustermaps in one store. Did you stop the newly created >> mon first? And I don't care about the ceph-mon.target, that's always >> on to ensure the MON starts automatically after boot. >> >> Can you clarify that you really have all the clustermaps in one store? >> If not, you'll need to repeat the steps. In theory the steps should >> work exactly as they're described. >> >> Zitat von Gilberto Ferreira <gilberto.nunes32@xxxxxxxxx>: >> >> > That's strange. >> > Now I have only the ceph-mon.target available: >> > >> > systemctl status ceph-mon.target >> > ● ceph-mon.target - ceph target allowing to start/stop all ceph-mon@ >> .service >> > instances at once >> > Loaded: loaded (/usr/lib/systemd/system/ceph-mon.target; enabled; >> > preset: enabled) >> > Active: active since Wed 2025-08-20 14:07:12 -03; 1min 47s ago >> > Invocation: 1fcbb21af715460294bd6d8549557ed9 >> > >> > Notice: journal has been rotated since unit was started, output may be >> > incomplete. >> > >> >>>> And you did rebuild the store from all OSDs as I mentioned, correct? >> > Yes... >> > Like that: >> > >> > ceph-volume lvm activate --all >> > mkdir /root/mon-store >> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 >> --no-mon-config >> > --op update-mon-db --mon-store-path mon-store/ >> > ceph-monstore-tool mon-store/ rebuild -- --keyring >> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 pve03 >> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ >> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp >> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ >> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db >> > >> > On each node. >> > --- >> > >> > >> > Gilberto Nunes Ferreira >> > +55 (47) 99676-7530 - Whatsapp / Telegram >> > >> > >> > >> > >> > >> > >> > Em qua., 20 de ago. de 2025 às 13:49, Eugen Block <eblock@xxxxxx> >> escreveu: >> > >> >> What does the monitor log? Does it at least start successfully? And >> >> you did rebuild the store from all OSDs as I mentioned, correct? >> >> >> >> Zitat von Gilberto Ferreira <gilberto.nunes32@xxxxxxxxx>: >> >> >> >> > Hi again... >> >> > I have reinstall all Proxmox nodes and install ceph on each node. >> >> > Create the mons and mgr on eatch node. >> >> > I have issue the command ceph-volume lvm activate --all, on each >> node, in >> >> > order bring up the /var/lib/ceph/osd/<node> >> >> > After that I ran this commands: >> >> > ceph-volume lvm activate --all >> >> > mkdir /root/mon-store >> >> > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 >> >> --no-mon-config >> >> > --op update-mon-db --mon-store-path mon-store/ >> >> > ceph-monstore-tool mon-store/ rebuild -- --keyring >> >> > /etc/pve/priv/ceph.client.admin.keyring --mon-ids pve01 pve02 pve03 >> >> > mv /var/lib/ceph/mon/ceph-pve01/store.db/ >> >> > /var/lib/ceph/mon/ceph-pve01/store.db-bkp >> >> > cp -rf mon-store/store.db/ /var/lib/ceph/mon/ceph-pve01/ >> >> > chown -R ceph:ceph /var/lib/ceph/mon/ceph-pve01/store.db >> >> > >> >> > But now I got nothing! >> >> > No monitor, no manager, no osd, none! >> >> > >> >> > Perhaps somebody point me what I did wrong. >> >> > >> >> > Thanks >> >> > >> >> > Em qua., 20 de ago. de 2025 às 11:32, Gilberto Ferreira < >> >> > gilberto.nunes32@xxxxxxxxx> escreveu: >> >> > >> >> >> I can see the content of the mentioned folders just after issue the >> >> >> command ceph-volume.... >> >> >> Thanks anyway. >> >> >> >> >> >> >> >> >> >> >> >> Em qua., 20 de ago. de 2025 às 11:26, Eugen Block <eblock@xxxxxx> >> >> >> escreveu: >> >> >> >> >> >>> I assume you're right. Do you see the OSD contents in >> >> >>> /var/lib/ceph/osd/ceph-pve01 after activating? >> >> >>> And remember to collect the clustermap from all OSDs for this >> >> >>> procedure to succeed. >> >> >>> >> >> >>> Zitat von Gilberto Ferreira <gilberto.nunes32@xxxxxxxxx>: >> >> >>> >> >> >>> > I see... >> >> >>> > >> >> >>> > But I had another problem. >> >> >>> > The script from (0) indicate that should be exist a >> /var/lib/ceph/osd >> >> >>> > folder, like: >> >> >>> > /var/lib/ceph/osd/ceph-pve01 >> >> >>> > /var/lib/ceph/osd/ceph-pve02 >> >> >>> > and so on. >> >> >>> > >> >> >>> > But this folder appears only if I run ceph-volume lvm activate >> --all. >> >> >>> > So my question is: when I should run this command: after or >> before >> >> use >> >> >>> the >> >> >>> > script? >> >> >>> > I think I need to run ceph-volume lvm activate --all, right? >> >> >>> > Just to clarify. >> >> >>> > >> >> >>> > Thanks >> >> >>> > >> >> >>> > Em qua., 20 de ago. de 2025 às 11:08, Eugen Block <eblock@xxxxxx >> > >> >> >>> escreveu: >> >> >>> > >> >> >>> >> Yes, you need a monitor. The mgr is not required and can be >> deployed >> >> >>> >> later. After you created the monitor, replace the mon store >> contents >> >> >>> >> by the collected clustermaps from the mentioned procedure. Keep >> the >> >> >>> >> ownerships of the directories/files in mind. If the monitor >> starts >> >> >>> >> successfully (with the original FSID), you can try to start one >> of >> >> the >> >> >>> >> OSDs. If that works, start the rest of them, wait for the >> peering >> >> >>> >> storm to settle, create two more monitors and two mgr daemons. >> >> >>> >> >> >> >>> >> Note that if you lose the mon store and you had a CephFS, you'll >> >> need >> >> >>> >> to recreate that from the existing pools. >> >> >>> >> >> >> >>> >> Zitat von Gilberto Ferreira <gilberto.nunes32@xxxxxxxxx>: >> >> >>> >> >> >> >>> >> > Hi >> >> >>> >> > >> >> >>> >> > Do I need to create any mon and/or mgr in the new ceph >> cluster? >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > >> >> >>> >> > Em seg., 18 de ago. de 2025 às 13:03, Eugen Block < >> eblock@xxxxxx> >> >> >>> >> escreveu: >> >> >>> >> > >> >> >>> >> >> Hi, >> >> >>> >> >> >> >> >>> >> >> this sounds like you created a new cluster (new fsid), the >> OSDs >> >> >>> still >> >> >>> >> >> have the previous fsid configured. I'd rather recommend to >> follow >> >> >>> this >> >> >>> >> >> procedure [0] to restore the mon store utilizing OSDs rather >> than >> >> >>> >> >> trying to manipulate otherwise intact OSDs to fit into the >> "new" >> >> >>> >> >> cluster. That way you'll have "your" cluster back. I don't >> know >> >> if >> >> >>> >> >> there are any specifics to using proxmox, though. But the >> >> mentioned >> >> >>> >> >> procedure seems to work just fine, I've read multiple >> reports on >> >> >>> this >> >> >>> >> >> list. Luckily, I haven't had to use it myself. >> >> >>> >> >> >> >> >>> >> >> Regards, >> >> >>> >> >> Eugen >> >> >>> >> >> >> >> >>> >> >> [0] >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >>> >> >> >> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#recovery-using-osds >> >> >>> >> >> >> >> >>> >> >> Zitat von Gilberto Ferreira <gilberto.nunes32@xxxxxxxxx>: >> >> >>> >> >> >> >> >>> >> >> > Hi >> >> >>> >> >> > >> >> >>> >> >> > I have 3 nodes Proxmox Cluster with CEPH, and after a >> crash, I >> >> >>> have to >> >> >>> >> >> > reinstall Proxmox from scratch, along with Ceph. >> >> >>> >> >> > OSD are intact. >> >> >>> >> >> > I already did ceph-volume lvm activate --all and the OSD >> >> appears >> >> >>> with >> >> >>> >> >> > ceph-volum lvm list and I got a folder with the name of >> the OSD >> >> >>> under >> >> >>> >> >> > /var/lib/ceph/osd. >> >> >>> >> >> > However is not appear in ceph osd tree or ceph -s or even >> in >> >> the >> >> >>> web >> >> >>> >> gui. >> >> >>> >> >> > Is there any way to re-add this OSD to Proxmox CEPH? >> >> >>> >> >> > >> >> >>> >> >> > Thanks a lot for any help. >> >> >>> >> >> > >> >> >>> >> >> > >> >> >>> >> >> > Best Regards >> >> >>> >> >> > --- >> >> >>> >> >> > Gilbert >> >> >>> >> >> > _______________________________________________ >> >> >>> >> >> > ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> _______________________________________________ >> >> >>> >> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >> >> >>> >> > _______________________________________________ >> >> >>> >> > ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >> >>> >> >> >> >>> >> _______________________________________________ >> >> >>> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >> >>> > _______________________________________________ >> >> >>> > ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >>> >> >> >>> _______________________________________________ >> >> >>> ceph-users mailing list -- ceph-users@xxxxxxx >> >> >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >>> >> >> >> >> >> > _______________________________________________ >> >> > ceph-users mailing list -- ceph-users@xxxxxxx >> >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> >> >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users@xxxxxxx >> >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users@xxxxxxx >> > To unsubscribe send an email to ceph-users-leave@xxxxxxx >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> > _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |