It’s a function of your use-case. > On Apr 14, 2025, at 8:41 AM, Anthony Fecarotta <anthony@xxxxxxxxxxx> wrote: > >> MDS (if you’re going to CephFS vs using S3 object storage or RBD block) > Hi Anthony, > > Can you elaborate on this remark? > > Should one choose between using CephFS vs S3 Storage (as it pertains to best practices)? > > On Proxmox, I am. using both CephFS and RBD. > > > Regards, > [image] > Anthony Fecarotta > Founder & President > [image] anthony@xxxxxxxxxxx <mailto:anthony@xxxxxxxxxxx> > [image] 224-339-1182 [image] (855) 625-0300 > [image] 1 Mid America Plz Flr 3 Oakbrook Terrace, IL 60181 > [image] www.linehaul.ai <http://www.linehaul.ai/> > [image] <http://www.linehaul.ai/> > [image] <https://www.linkedin.com/in/anthony-fec/> > > On Sun Apr 13, 2025, 04:28 PM GMT, Anthony D'Atri <mailto:anthony.datri@xxxxxxxxx> wrote: >> >>> On Apr 13, 2025, at 12:00 PM, Brendon Baumgartner <brendon@xxxxxxxxxx> wrote: >>> >>> >>>> On Apr 11, 2025, at 10:13, gagan tiwari <gagan.tiwari@xxxxxxxxxxxxxxxxxx> wrote: >>>> >>>> Hi Anthony, >>>> We will be using Samsung SSD 870 QVO 8TB disks on >>>> all OSD servers. >>> >>> I’m a newbie to ceph and I have a 4 node cluster and it doesn’t have a lot of users so downtime is easily scheduled for tinkering. I started with consumer SSDs (SATA/NVMEs) because they were free and lying around. Performance was bad. Then just the NVMEs, still bad. Then enterprise SSDs, still bad (relative to DAS anyway). >> >> Real enteprise SSDs? Enterprise NVMe not enterprise SATA? Sellers can lie sometimes. Also be sure to update firmware to the latest, that can make a substantial difference. >> >> Other factors include: >> >> * Enough hosts and OSDs. Three hosts with one OSD each aren’t going to deliver a great experience >> * At least 6GB of available physmem per NVMe OSD >> * How you measure - a 1K QD1 fsync workload is going to be more demanding than a buffered 64K QD32 workload. >>> >>> Each step on the journey to enterprise SSDs made things faster. The problem with the consumer stuff is the latency. Enterprise SSDs are 0-2ms. Consumer SSDs are 15-300ms. As you can see, the latency difference is significant. >> >> Some client SSDs are “DRAMless”, they don’t use ~~ 1GB of onboard RAM per 1TB of capacity as the LBA indirection table. This can be a substantial issue for enterprise workloads. >> >>> >>> So from my experience, I would say ceph is very slow in general compared to DAS. You need all the help you can get. >>> >>> If you want to use the consumer stuff, I would recommend to make a slow tier (2nd pool with a different policy). Or I suppose just expect it to be slow in general. I still have my consumer drives installed, just configured as a 2nd tier which is unused right now because we have an old JBOD for 2nd tier that is much faster. >> >> How much drives in each? >>> >>> Good luck! >>> >>> _BB >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |