Ceph Notes

I use ceph with my three node Proxmox VE cluster. This document is to record any sort of ceph things I do outside of the vanilla config.

When I originally built it, each node had four 4 TB SAS HDDs in it (specifically HGST HUS724040ALS640). This was OK, but not exactly fast. After doing some research, I decided it would be worth ditching one HDD per node to make room for an SSD DB+WAL disk (which I chose the HGST HUSMM1640ASS200 for). This involved removing one OSD in each node, installing the SSD in its place, then removing and re-adding the remaining OSDs one by one. There are ways to migrate the DB and WAL to a fast SSD, but I didn't trust myself with that.

There weren't really any step-by-step examples that I could find at the time, so I wanted to document the process I followed to achieve this. It worked great, just took about a week to get all the shuffling done, so patience was required.

First, some tweaks in an attempt to make recovery in between OSD removals/additions faster:

ceph config set osd osd_max_backfills 8 (more simultaneous backfill streams per OSD)
ceph config set osd osd_recovery_max_active 12 (more simultaneous recovery threads per OSD)
ceph config set osd osd_recovery_sleep 0.0 (remove any delay between recover operations)
ceph config set osd osd_recovery_op_priority 63 (give recovery work maximum queue priority)

Now rinse and repeat through each OSD:

1. Mark the target OSD out via ceph osd out [OSD ID]. Ceph will start rebuilding its placement groups (PGs) on other disks.
2. Monitor ceph cluster health via ceph -s and wait until all PGs are active+clean.
3. Once all PGs are active+clean, stop the OSD daemon via systemctl stop ceph-osd@[OSD ID].
4. Run pveceph osd destroy [OSD ID] --cleanup to remove the OSD from the cluster and to clean the disk for re-use.
5. Re-deploy the OSD specifying the fast disk for DB+WAL by running pveceph osd create /dev/sdX --db_dev /dev/sdY --db_dev_size 102 --encrypted where sdX is the OSD device and sdY is the SSD you're using for DB+WAL. Be sure to calculate the appropriate db_dev_size for your environment. Since my OSDs are 4 TB, a 102 GB DB+WAL volume is sufficient if going by the 2.5% rule, but your environment may be different.
6. Wait until the cluster has fully backfilled/rebalanced before moving on to the next OSD.

Once all OSD shuffling is complete, we reset the tweaks we made before starting:

ceph config rm osd osd_max_backfills
ceph config rm osd osd_recovery_max_active
ceph config rm osd osd_recovery_sleep
ceph config rm osd osd_recovery_op_priority