Ceph cross-node networking #2

fnetX commented 2023年02月17日 16:41:39 +01:00

We are considering to add a second server to our Ceph cluster. We are currently connected with 2x1Gbps to our provider's switch, which is likely not enough for Ceph long-term.

How best can we realize dedicated networking between the machines, also considering how changes can be made with minimum downtime (e.g. installing new network cards into our server).

We are considering to add a second server to our Ceph cluster. We are currently connected with 2x1Gbps to our provider's switch, which is likely not enough for Ceph long-term. How best can we realize dedicated networking between the machines, also considering how changes can be made with minimum downtime (e.g. installing new network cards into our server).

iQ commented 2023年02月20日 01:34:34 +01:00

As said in the other ticket: Depending on the Budget ...

I would really suggest going for at least a 3node cluster instead of two.
And yes, 1Gbps is not enough. But nowadays 10G is in viable reach for cheap.
If possible install 10G cards and even a cheap 10G switch (I don't suggest a cheap switch) will be doing the job mostly.

having at least a 10G cluster network is neccessary, before even considering multiple nodes.

As said in the other ticket: Depending on the Budget ... I would really suggest going for at least a 3node cluster instead of two. And yes, 1Gbps is not enough. But nowadays 10G is in viable reach for cheap. If possible install 10G cards and even a cheap 10G switch (I don't suggest a cheap switch) will be doing the job mostly. having at least a 10G cluster network is neccessary, before even considering multiple nodes.

cardes commented 2023年02月23日 20:40:21 +01:00

If i may add some points for consideration:

• 3 nodes are minimum for production, ceph is not that reliable if you run into bottlenecks so you really need that quorum for recovery
• if you only use HDDs and do not plan to invest in switchs, you can use a 3 node mesh setup like mentioned here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server This is also more reliable than a single point of failure switch. If you have the financial leeway of course stacked switches with LACP are the best option.
• separating the ceph networks in public and cluster network helps with reliability, also those networks should at least use their nics exclusively.

I hope that helps a bit and good luck with your setup.

If i may add some points for consideration: - 3 nodes are minimum for production, ceph is not that reliable if you run into bottlenecks so you really need that quorum for recovery - if you only use HDDs and do not plan to invest in switchs, you can use a 3 node mesh setup like mentioned here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server This is also more reliable than a single point of failure switch. If you have the financial leeway of course stacked switches with LACP are the best option. - separating the ceph networks in public and cluster network helps with reliability, also those networks should at least use their nics exclusively. I hope that helps a bit and good luck with your setup.

fnetX commented 2023年02月23日 21:41:39 +01:00

Thank you for the comments. The idea to spin up a three-node mesh network sounds the simplest for the near future, which could later be expanded into a 4+ node switched network.

Thank you for the comments. The idea to spin up a three-node mesh network sounds the simplest for the near future, which could later be expanded into a 4+ node switched network.

yoctozepto commented 2023年08月30日 14:28:11 +02:00

I have recently analysed the card and switch market and found 25G to be the sweet spot (best performance per euro).

Regarding networking for Ceph, it can really help to have separate cluster and client (aka public but it's a big misnomer) networks. This way they don't impact each other and it's easier to measure the saturation and estimate the growth needs.

Always have multiples of 3 as the number of Ceph nodes and put them equally into different racks - to eliminate the risk of having one rack shut down the whole cluster: the racks usually have separate power sources, at least to some degree.

(Also, always have a backup somewhere else. :D )

I have recently analysed the card and switch market and found 25G to be the sweet spot (best performance per euro). Regarding networking for Ceph, it can really help to have separate cluster and client (aka public but it's a big misnomer) networks. This way they don't impact each other and it's easier to measure the saturation and estimate the growth needs. Always have multiples of 3 as the number of Ceph nodes and put them equally into different racks - to eliminate the risk of having one rack shut down the whole cluster: the racks usually have separate power sources, at least to some degree. (Also, always have a backup somewhere else. :D )