Ceph Roadmap #1

fnetX commented 2023年02月17日 16:38:45 +01:00

Discussion: How best to continue expanding our Ceph cluster

Current setup as of 2023年02月17日:

• single server holds Ceph data
• 3x OSD on 7TiB HDD partitions each (three independent disks, other partition no longer in use)
• 2x OSD with 7TiB SSD devices
• data stored on a single CephFS volume
• there are 6x3.5" SATA slots unused in the server
• two 2.5" SATA SSDs (7TiB) are spare and not currently installed in any server

Questions:

• In order to speed up the system, should we immediately remove some HDD (and keep one, or two), or rather keep the many devices to spread IO?
• Mid-term, should we rather deploy a second server with more SSDs, or add more SSDs to the current machine?

Discussion: How best to continue expanding our Ceph cluster Current setup as of 2023年02月17日: - single server holds Ceph data - 3x OSD on 7TiB HDD partitions each (three independent disks, other partition no longer in use) - 2x OSD with 7TiB SSD devices - data stored on a single CephFS volume - there are 6x3.5" SATA slots unused in the server - two 2.5" SATA SSDs (7TiB) are spare and not currently installed in any server Questions: - In order to speed up the system, should we immediately remove some HDD (and keep one, or two), or rather keep the many devices to spread IO? - Mid-term, should we rather deploy a second server with more SSDs, or add more SSDs to the current machine?

fnetX commented 2023年02月17日 16:39:45 +01:00

Other options could be installing SSDs as NVMe cards eventually. I don't know the exact requirements, though.

Other options could be installing SSDs as NVMe cards eventually. I don't know the exact requirements, though.

dmoonfire commented 2023年02月20日 01:28:45 +01:00

From my own experience, getting Ceph to at least three separate servers is a good thing. My own network worked out much smoother when I was able to spread out the load (and Ceph really likes having at least three servers for redundancy, more so when you should have roughly an odd number of mons). Also, you didn't list RAM on the servers but you need about 1 GB/1 TB of OSD, otherwise the server ends up grinding too much on the drives. Between those two things, I got a much smoother experience with a pair of 8 TB/machine each on three 16 GB machines.

The second bit would be (as mentioned in another issue) would be to work on a SSD/NVMe pool. Start by moving your metadata pools over to it. Since the metadata is what is looked up the most, having that faster helps.

Thirdly, having a dedicated network just between your Ceph machines for cross-talk can help with the traffic as they are constantly moving bits around each other for rebalancing and moving drives. At a smaller size, this is less of an issue, but if you set it up so you can just add OSDs over time, it will start get more of an issue. I just can't tell you when it will become a problem.

And to answer the question, I think you should balance out your HDD and SDDs across both machines as much as possible.

From my own experience, getting Ceph to at least three separate servers is a good thing. My own network worked out much smoother when I was able to spread out the load (and Ceph really likes having at least three servers for redundancy, more so when you should have roughly an odd number of mons). Also, you didn't list RAM on the servers but you need about 1 GB/1 TB of OSD, otherwise the server ends up grinding too much on the drives. Between those two things, I got a much smoother experience with a pair of 8 TB/machine each on three 16 GB machines. The second bit would be (as mentioned in another issue) would be to work on a SSD/NVMe pool. Start by moving your metadata pools over to it. Since the metadata is what is looked up the most, having that faster helps. Thirdly, having a dedicated network just between your Ceph machines for cross-talk can help with the traffic as they are constantly moving bits around each other for rebalancing and moving drives. At a smaller size, this is less of an issue, but if you set it up so you can just add OSDs over time, it will start get more of an issue. I just can't tell you *when* it will become a problem. And to answer the question, I think you should balance out your HDD and SDDs across both machines as much as possible.

iQ commented 2023年02月20日 01:30:37 +01:00

Depending on the Budget mostly.

I have not really an idea of the traffic/IO patterns you currently have to work with.
It mostly depends on the size of the working set and if this fits on the available "fast" storage.

But: I would advocate for building a pool of SSDs to keep the latency down. HDDs are to slow for that. Even one SATA SSD alone has 10x lower latency than a HDD, so even with 10HDDs you couldn't get to these numbers.

You don't have to throw away the HDDs but take them out of the active pool. You can use them for Backup/Archive stuff, and maybe even save some capacity by choosing Erasure Coding for redundancy.

Resilience wise I would go for at least a three node setup and try to distribute the available devices across these. Having only one Machine makes it a SPoF (of course).
There you can also setup other MON services to benefit.

The nice thing is, that most of the work can be done without taking the CEPH down.
You can add additional OSDs on the fly, rebalance, and drain the HDDs from the pool.

Having some numbers and ceph stats could help to identify the most pressing bottlenecks.
I would assume its the HDDs latency.

BTW: If you have SSDs then they will still be slow as SSDs even when put as NVMe via adapter.
If they are NVMe in an SSD casing, then of course, you will benefit massively from the latencyboost of them in an NVMe Slot, and you should consider putting the WAL and the DB onto these ...

Depending on the Budget mostly. I have not really an idea of the traffic/IO patterns you currently have to work with. It mostly depends on the size of the working set and if this fits on the available "fast" storage. But: I would advocate for building a pool of SSDs to keep the latency down. HDDs are to slow for that. Even one SATA SSD alone has 10x lower latency than a HDD, so even with 10HDDs you couldn't get to these numbers. You don't have to throw away the HDDs but take them out of the active pool. You can use them for Backup/Archive stuff, and maybe even save some capacity by choosing Erasure Coding for redundancy. Resilience wise I would go for at least a three node setup and try to distribute the available devices across these. Having only one Machine makes it a SPoF (of course). There you can also setup other MON services to benefit. The nice thing is, that most of the work can be done without taking the CEPH down. You can add additional OSDs on the fly, rebalance, and drain the HDDs from the pool. Having some numbers and ceph stats could help to identify the most pressing bottlenecks. I would assume its the HDDs latency. BTW: If you have SSDs then they will still be slow as SSDs even when put as NVMe via adapter. If they are NVMe in an SSD casing, then of course, you will benefit massively from the latencyboost of them in an NVMe Slot, and you should consider putting the WAL and the DB onto these ...

yoctozepto commented 2023年08月30日 14:36:05 +02:00

If you run SSDs and HDDs plainly in the same pool, then you are likely undermining the performance of SSDs a lot.

Do you have data duplication or triplication enabled? (I hope you have at least duplication, but it is much safer with triplication).

Also note that cheap, non-server-class SSDs are bad juju - they can't hold the load the HDDs can and might experience a sudden death taking the data with themselves. ;-)

I am open to have a chat to learn more about the current status, needs and so on. @fnetX

If you run SSDs and HDDs plainly in the same pool, then you are likely undermining the performance of SSDs a lot. Do you have data duplication or triplication enabled? (I hope you have at least duplication, but it is much safer with triplication). Also note that cheap, non-server-class SSDs are bad juju - they can't hold the load the HDDs can and might experience a sudden death taking the data with themselves. ;-) I am open to have a chat to learn more about the current status, needs and so on. @fnetX

fnetX commented 2023年08月30日 18:40:50 +02:00

The current setup is the following:

We have three HGST 12TB HDD and 2x Samsung MZ7L37T6HBLA if I recall correctly.

We adjusted the Ceph primary affinity to shift the read load to the SSDs.

All pools have x3 replication at the moment.

We have a second server (not yet in production) with 8 2.5" disk slots, and bought some 4TB Notebook HDD some time ago, because back then it was considered the cheapest strategy to "just use them". It doesn't sound like a wise option now, but maybe combined with the primary affinity they can still be used for cheap redundancy.

The current setup is the following: We have three HGST 12TB HDD and 2x Samsung MZ7L37T6HBLA if I recall correctly. We adjusted the Ceph primary affinity to shift the read load to the SSDs. All pools have x3 replication at the moment. We have a second server (not yet in production) with 8 2.5" disk slots, and bought some 4TB Notebook HDD some time ago, because back then it was considered the cheapest strategy to "just use them". It doesn't sound like a wise option now, but maybe combined with the primary affinity they can still be used for cheap redundancy.

yoctozepto commented 2023年08月30日 19:01:13 +02:00

Thanks for the update.

Hmm, these SSDs are server-grade at least, which is good, yet they seem to be read-optimised (only 1 DWPD, where for write-optimised it is considered to be >10 DWPD; DWPD - disk writes per day, over the 5-year term).
SMART monitoring should help with predicting when the failure might happen.

Good that you have adjusted the affinity. I originally missed that your are trying to optimise reads, not writes. That's a good move.

Great that you have triplication. It would be good to check how PGs got mapped to the OSDs - is it the case that each PG is on an SSD and then two HDDs? That would be the best situation with your setup. I think it might not be the case because of the weights; you are putting roughly 2x7=14 vs 3x7=21 which means HDDs should be more likely to get PGs. Not sure what ceph does if you, for example, set the affinity of HDDs to zero. I would assume that then you kind of get the cap at 14 as there are no other primary candidates for PGs to be allocated...

How saturated is your ceph cluster now?

The laptop 2,5" disks with 5.4k are going to be latency bumpers. Sadly, with 2,5" it's hard to get decent GB/euro ratios when you are trying to optimise for maximum capacity.
That said, if only reads are to be optimised, you might be in luck but one needs to tread carefully now to ensure a satisfying PG allocation.
The theory says that then you would need 1/3 of the storage space to be served by SSDs anyways.
Then, you might be able to also balance the latency by having (on average) fewer PGs on slower disks than on faster disks (as they are also larger).

Thanks for the update. Hmm, these SSDs are server-grade at least, which is good, yet they seem to be read-optimised (only 1 DWPD, where for write-optimised it is considered to be >10 DWPD; DWPD - disk writes per day, over the 5-year term). SMART monitoring should help with predicting when the failure might happen. Good that you have adjusted the affinity. I originally missed that your are trying to optimise reads, not writes. That's a good move. Great that you have triplication. It would be good to check how PGs got mapped to the OSDs - is it the case that each PG is on an SSD and then two HDDs? That would be the best situation with your setup. I think it might not be the case because of the weights; you are putting roughly 2x7=14 vs 3x7=21 which means HDDs should be more likely to get PGs. Not sure what ceph does if you, for example, set the affinity of HDDs to zero. I would assume that then you kind of get the cap at 14 as there are no other primary candidates for PGs to be allocated... How saturated is your ceph cluster now? The laptop 2,5" disks with 5.4k are going to be latency bumpers. Sadly, with 2,5" it's hard to get decent GB/euro ratios when you are trying to optimise for maximum capacity. That said, if only reads are to be optimised, you might be in luck but one needs to tread carefully now to ensure a satisfying PG allocation. The theory says that then you would need 1/3 of the storage space to be served by SSDs anyways. Then, you might be able to also balance the latency by having (on average) fewer PGs on slower disks than on faster disks (as they are also larger).

fnetX commented 2023年08月30日 19:06:11 +02:00

We are really optimizing for reads.

Let's have a look at the stats for the SSDs:

kB_read/s kB_wrtn/s
111.7M 348.2k 
80.0M 420.3k

We are really optimizing for reads. Let's have a look at the stats for the SSDs: ~~~ kB_read/s kB_wrtn/s 111.7M 348.2k 80.0M 420.3k ~~~