Backup solutions #5

fnetX commented 2023年02月19日 21:49:15 +01:00

In Codeberg-Infrastructure/planning#4 we tracked the creation of a new backup solution.

The current state is that we rsync-mirror updated repositories into an offsite location, and we are improving that experiment.

However, if someone has experience to suggest a more viable solution for our current tech stack (CephFS) and / or might even be available to produce them, please feel free to comment.

In https://codeberg.org/Codeberg-Infrastructure/planning/issues/4 we tracked the creation of a new backup solution. The current state is that we rsync-mirror updated repositories into an offsite location, and we are improving that experiment. However, if someone has experience to suggest a more viable solution for our current tech stack (CephFS) and / or might even be available to produce them, please feel free to comment.

iQ commented 2023年02月20日 01:45:35 +01:00

with ceph you can mirror a pool between two ceph clusters. so having an backup/archive ceph-cluster would then ease the backup process.

Because rsync is Filebased, it will iterate over the whole file tree and trash all your caches while reading through everything. the Ceph-Pool-Mirror approach works on a lower level and only transfers the changed blocks.

with ceph you can mirror a pool between two ceph clusters. so having an backup/archive ceph-cluster would then ease the backup process. Because rsync is Filebased, it will iterate over the whole file tree and trash all your caches while reading through everything. the Ceph-Pool-Mirror approach works on a lower level and only transfers the changed blocks.

fnetX commented 2023年04月04日 13:18:36 +02:00

@iQ do you have some references at hand on how to get started? I didn't yet find some information on how we could mirror the CephFS (data and meta) pools.

@iQ do you have some references at hand on how to get started? I didn't yet find some information on how we could mirror the CephFS (data and meta) pools.

JavierVilarroig commented 2023年08月22日 23:04:41 +02:00

I can take care of deploying and maintaining Proxmox Backup based solution for data survivability.

For data availability the type of mirror proposed by @iQ sounds OK for guaranteeing data availability, but PBS can still be used for survivability.

I do it already in my daily work (1.5T data). I have a setup based in:

• A main PBS server used for storing the backups
• A second PBS server mirroring the backups from the main
• A third copy (encrypted) stored in a couple of external USB btrfs raid1 drives

If interested, we can discuss the details in Matrix.

I can take care of deploying and maintaining Proxmox Backup based solution for data survivability. For data availability the type of mirror proposed by @iQ sounds OK for guaranteeing data availability, but PBS can still be used for survivability. I do it already in my daily work (1.5T data). I have a setup based in: - A main PBS server used for storing the backups - A second PBS server mirroring the backups from the main - A third copy (encrypted) stored in a couple of external USB btrfs raid1 drives If interested, we can discuss the details in Matrix.

AfroThundr commented 2024年04月15日 01:50:58 +02:00

@iQ do you have some references at hand on how to get started? I didn't yet find some information on how we could mirror the CephFS (data and meta) pools.

Depending on your usage, either CephFS Mirroring or RBD Mirroring was probably meant here.

> @iQ do you have some references at hand on how to get started? I didn't yet find some information on how we could mirror the CephFS (data and meta) pools. Depending on your usage, either [CephFS Mirroring](https://docs.ceph.com/en/latest/dev/cephfs-mirroring/#) or [RBD Mirroring](https://docs.ceph.com/en/latest/rbd/rbd-mirroring/#) was probably meant here.