Possibilities to split CephFS pool for HDD and SSD based on use case #3

fnetX commented 2023年02月17日 16:44:38 +01:00

Currently, we have one CephFS pool holding all Forgejo data (Git files, attachments, avatars, LFS, ...).

Because HDD have proven to slow for many small file accesses for Git operations, we are adding SSDs to the cluster so that the data is migrated to faster storage.

Because the SSDs are very costly, it might make sense to split the data based on use case, and have the Git files on SSDs while the "normal" files stay on the HDD (for larger linear reads, HDDs are likely okay).

Can this be done with CephFS? Can Ceph caching reach a similar effect (speeding up Git operations while keeping other data on the HDD)?

Currently, we have one CephFS pool holding all Forgejo data (Git files, attachments, avatars, LFS, ...). Because HDD have proven to slow for many small file accesses for Git operations, we are adding SSDs to the cluster so that the data is migrated to faster storage. Because the SSDs are very costly, it might make sense to split the data based on use case, and have the Git files on SSDs while the "normal" files stay on the HDD (for larger linear reads, HDDs are likely okay). Can this be done with CephFS? Can Ceph caching reach a similar effect (speeding up Git operations while keeping other data on the HDD)?

iQ commented 2023年02月20日 01:41:34 +01:00

Ceph caching can help in parts. But yes, splitting the data between write-heavy on SSD and mostly-read on HDD is possible. The simplest solution would be to have multiple CephFS on different Pools that are mounted to different locations for the app..

Another question, depending on the setup, is, if you really need a CephFS as a shared storage, or if it would be enough to supply a blockdevice via Rados to the APP.

I don't know enough of forgejo yet, if it makes sense in some clustering scenario, or if all app instances need a shared filesystem for syncronization.

Ceph caching can help in parts. But yes, splitting the data between write-heavy on SSD and mostly-read on HDD is possible. The simplest solution would be to have multiple CephFS on different Pools that are mounted to different locations for the app.. Another question, depending on the setup, is, if you really need a CephFS as a shared storage, or if it would be enough to supply a blockdevice via Rados to the APP. I don't know enough of forgejo yet, if it makes sense in some clustering scenario, or if all app instances need a shared filesystem for syncronization.

fnetX commented 2023年04月04日 16:04:43 +02:00

Well, the idea of using CephFS was to spin up multiple nodes. It is quite surprising that we got so far with only a single server, and yes for HA it would be enough. However, this would require another quite heavy data migration.

Also, I estimate that we should soon look into running multiple instances of Forgejo, within the next 12 to 15 months, or we would need to look into some other scaling method (like serving assets from another machine etc). I'm not sure what is easy to achieve with Forgejo.

Still, since we installed SSDs, CephFS performs pretty well. We're still running on a dirty (SSD + HDD mixed) cluster and it is going well. So I suppose that splitting this based on access patterns (SSD-only for Git, and HDD-only for regular files for cost saving) sounds like a good idea?

Well, the idea of using CephFS was to spin up multiple nodes. It is quite surprising that we got so far with only a single server, and yes for HA it would be enough. However, this would require another quite heavy data migration. Also, I estimate that we should soon look into running multiple instances of Forgejo, within the next 12 to 15 months, or we would need to look into some other scaling method (like serving assets from another machine etc). I'm not sure what is easy to achieve with Forgejo. Still, since we installed SSDs, CephFS performs pretty well. We're still running on a dirty (SSD + HDD mixed) cluster and it is going well. So I suppose that splitting this based on access patterns (SSD-only for Git, and HDD-only for regular files for cost saving) sounds like a good idea?

yoctozepto commented 2023年08月30日 14:46:00 +02:00

CephFS supports having data pools per-directory: https://docs.ceph.com/en/quincy/cephfs/file-layouts/#adding-data-pool-to-file-system and it would make sense to have git stuff on SSDs and HDDs for everything else.

(As mentioned in the other thread, I really hope that at least duplication is happening and these are server-grade SSDs optimised for writes).

CephFS supports having data pools per-directory: https://docs.ceph.com/en/quincy/cephfs/file-layouts/#adding-data-pool-to-file-system and it would make sense to have git stuff on SSDs and HDDs for everything else. (As mentioned in the other thread, I really hope that at least duplication is happening and these are server-grade SSDs optimised for writes).