New Backup Solution #4

fnetX commented 2023年01月06日 02:00:50 +01:00

I think we need a new backup solution.

Current Situation

• Gitea data / attachments stored on CephFS
  • single node / small Ceph cluster, 1.5TB of data
• Database (MariaDB) and container filesystems stored on BTRFS SSDs
  • (no Ceph involved)
  • < 200 GB btrfs data
• two Borg backup job scheduled regularly
  • encrypted offsite backup
  • independent runs for BTRFS and Ceph data
  • regular cleanup jobs
  • Ceph job frequently takes > 10 hours to complete
  • io / cpu load of backup seems acceptable

Problems

Since the Borg backup job seems to take much more time than it did in the past, this appears like an upcoming scaling problem to me.

Borg sometimes fails by keeping the lock infinitely which requires manual fixup (I think we observed this for Codeberg 4 times over the course of a year). Additionally, we had some broken state after a run was interrupted which required some check/repair operation.

Borg locks the repo, so there are no concurrent jobs. This is not a problem for the backup jobs, but for cleanup / repair operations, this locks the repo for quite a while.

Considerations

CephFS is likely prone to some increased i/o latency on our system, especially since we're already having i/o trouble in #2.

Most Ceph backup solutions work with disk images and snapshots instead. Maybe we should look into snapshotting this, too? https://docs.ceph.com/en/latest/dev/cephfs-snapshots/

And combine with something else than borg?

I've been watching talks and reading suggestions lately. Most interesting to me sounds to have a standby Ceph cluster somewhere else, though I'm really not sure how to achieve this. Also, this would be more of a long-term project and not a short-time measure wiht the resources we have.

I think we need a new backup solution. **Current Situation** - Gitea data / attachments stored on CephFS - single node / small Ceph cluster, 1.5TB of data - Database (MariaDB) and container filesystems stored on BTRFS SSDs - (no Ceph involved) - < 200 GB btrfs data - two Borg backup job scheduled regularly - encrypted offsite backup - independent runs for BTRFS and Ceph data - regular cleanup jobs - Ceph job frequently takes > 10 hours to complete - io / cpu load of backup seems acceptable **Problems** Since the Borg backup job seems to take much more time than it did in the past, this appears like an upcoming scaling problem to me. Borg sometimes fails by keeping the lock infinitely which requires manual fixup (I think we observed this for Codeberg 4 times over the course of a year). Additionally, we had some broken state after a run was interrupted which required some check/repair operation. Borg locks the repo, so there are no concurrent jobs. This is not a problem for the backup jobs, but for cleanup / repair operations, this locks the repo for quite a while. **Considerations** CephFS is likely prone to some increased i/o latency on our system, especially since we're already having i/o trouble in #2. Most Ceph backup solutions work with disk images and snapshots instead. Maybe we should look into snapshotting this, too? https://docs.ceph.com/en/latest/dev/cephfs-snapshots/ And combine with something else than borg? I've been watching talks and reading suggestions lately. Most interesting to me sounds to have a standby Ceph cluster somewhere else, though I'm really not sure how to achieve this. Also, this would be more of a long-term project and not a short-time measure wiht the resources we have.

fnetX commented 2023年01月16日 13:38:39 +01:00

The problem becomes more urgent. The current backup system worsened the IO issues we had in the past days a lot. Stopping the backup significantly speeds up Codeberg, and the io problem is causing the backup to take way too long.

We could try Restic backup and see if it performs better: https://restic.readthedocs.io/en/stable/

---

My proposed alternative is the following:

Concept:

• an offsite machine holds a complete mirror of our data
• a cronjob syncs only the latest changes to this machine (could be rsync, probably faster than crawling through all the data)
  • could reuse the approach for our garbage collection using find -mtime to check for repos that have been recently modified, or the Gitea API
• filesystem snapshots on the remote machine allow for data versioning
• open question: handling of renames and deleted repos (probably needs some hook in Gitea, or sync the list of repo names and remove missing)

Realization:

• some used disks in a RAID could be a (financially and ecological) sustainable solution
• using BTRFS as filesystem allows for FS snapshots and compression (not sure if it makes sense for Git repos, though); eventually even deduplication
• using BTRFS, we could use a RAID-1, because BTRFS has internal checksums
• to save energy, we could try to suspend the machine to RAM and wake it up in intervals
• to have a working suspend feature and re-use existing hardware, eventually a used desktop PC is enough for this job (often has about 6 HDD slots, which should be enough for about 15TB - 30TB of data)
• I already use a quite similar setup at home (using an office / non-19"-server which supports standby very well and has integrated remote management), using BTRFS and a bunch of old disks.

The problem becomes more urgent. The current backup system worsened the IO issues we had in the past days *a lot*. Stopping the backup significantly speeds up Codeberg, and the io problem is causing the backup to take way too long. We could try Restic backup and see if it performs better: https://restic.readthedocs.io/en/stable/ --- My proposed alternative is the following: Concept: - an offsite machine holds a complete mirror of our data - a cronjob syncs only the latest changes to this machine (could be rsync, probably faster than crawling through all the data) - could reuse the approach for our garbage collection using find -mtime to check for repos that have been recently modified, or the Gitea API - filesystem snapshots on the remote machine allow for data versioning - open question: handling of renames and deleted repos (probably needs some hook in Gitea, or sync the list of repo names and remove missing) Realization: - some used disks in a RAID could be a (financially and ecological) sustainable solution - using BTRFS as filesystem allows for FS snapshots and compression (not sure if it makes sense for Git repos, though); eventually even deduplication - using BTRFS, we could use a RAID-1, because BTRFS has internal checksums - to save energy, we could try to suspend the machine to RAM and wake it up in intervals - to have a working suspend feature and re-use existing hardware, eventually a used desktop PC is enough for this job (often has about 6 HDD slots, which should be enough for about 15TB - 30TB of data) - I already use a quite similar setup at home (using an office / non-19"-server which supports standby very well and has integrated remote management), using BTRFS and a bunch of old disks.

fsologureng commented 2023年01月16日 15:17:35 +01:00

Rsync and find -mtime are both file cache killers. I would use it with care.

Can I ask why BTRFS? Just techno curiosity

---

update: fs snapshots seems to me more suitable because it creates duplication of the data on write demand so it is easier to achieve data consistency. With rsync or find you would never achieve a good snapshot.

Rsync and find -mtime are both file cache killers. I would use it with care. Can I ask why BTRFS? Just techno curiosity --- update: fs snapshots seems to me more suitable because it creates duplication of the data on write demand so it is easier to achieve data consistency. With rsync or find you would never achieve a good snapshot.

fnetX commented 2023年01月16日 16:46:14 +01:00

I can't think of a file-based backup solution that is not a cache killer. My proposed workflow is: Instead of letting a backup solution look at the timestamps and sizes of every file, we only look at the mtime of the folders (much less) and then sync those that have recently changed. If we only compare the tiny files of the folders that have recently changed, chances are high they are still in cache, or might anyway be re-accessed (by users viewing the updates, creating a PR etc).

Since we aren't using disk images, I can't think of an easy way to manage backups. If you know of any backup solution that works well with CephFS (e.g. diff streaming), please let us know!

The mirror would basically be always up2date with our production data (maybe with a short delay), and filesystem snapshots are used on the remote machine to allow for some snapshot consistency.

We can also use snapshots on CephFS, but if we update repos one-by-one, I think it's unlikely to be too bad. CephFS snapshots likely also decrease performance, but we could look into enabling them per repository to ensure consistency for the rsync operation.

BTRFS: We can also use different methods. I'm more familiar with ext4 and it is faster, however the data is less protected in raid-1, and the features mentioned before are not available.
XFS, ZFS and the like might also be options, in the long term feature even an offsite Ceph cluster if necessary.
BTRFS is just something with features that I am currently getting familiar, and I think we should keep the number of different tools in the techstack small.

I can't think of a file-based backup solution that is not a cache killer. My proposed workflow is: Instead of letting a backup solution look at the timestamps and sizes of every file, we only look at the mtime of the folders (much less) and then sync those that have recently changed. If we only compare the tiny files of the folders that have recently changed, chances are high they are still in cache, or might anyway be re-accessed (by users viewing the updates, creating a PR etc). Since we aren't using disk images, I can't think of an easy way to manage backups. If you know of any backup solution that works well with CephFS (e.g. diff streaming), please let us know! The mirror would basically be always up2date with our production data (maybe with a short delay), and filesystem snapshots are used on the remote machine to allow for some snapshot consistency. We can also use snapshots on CephFS, but if we update repos one-by-one, I think it's unlikely to be too bad. CephFS snapshots likely also decrease performance, but we could look into enabling them per repository to ensure consistency for the rsync operation. BTRFS: We can also use different methods. I'm more familiar with ext4 and it is faster, however the data is less protected in raid-1, and the features mentioned before are not available. XFS, ZFS and the like might also be options, in the long term feature even an offsite Ceph cluster if necessary. BTRFS is just something with features that I am currently getting familiar, and I think we should keep the number of different tools in the techstack small.

gapodo commented 2023年01月16日 22:12:04 +01:00

There are better and worse ways in which backup applications traverse / search for changes in file systems, but in general file based backup is expensive period

(virtual) disk backups usually deal with backups using some form of change block tracking (CBT, please don't google CBT, I've made that mistake once... use change block tracking as the search term) also called dirty bit map (remembering which blocks were changed), which is comparatively efficient, if data isn't modified much / only metadata is touched (and the file system stores metadata separately). This is implemented in different ways, though many of the solutions rely on some form of snapshots to achieve this.

Snapshoting data no matter if on file level or a disk (the disk is also just a file, handled as if it was many small files for tracking) will create some performance penalties as it's either COW (copy on write) or COR (copy on read) unless CBT is used.

Some CBT mechanisms work around the COW/COR issue by only ever storing the latest data and just noting, that it has changed (CBT) since the last backup, which allows them to be more efficient when it comes to the impact of changes (though the tracking and checking also takes some performance, but mostly CPU not IO).

From a solution perspective, there is no magic solution to the issue, there are tradeoffs between IO, CPU, memory, bandwidth and data consistency (consistency is where snapshots and CBT shine).

I've extensively used Borg for a while and must say, that I've started turning away from it due to the already mentioned issue with repo locking (as well as inefficient tracking).

File Systems

I cannot really comment on the "which file system to use" part from my work background, so what I'm commenting here is my ~20TB net home environment, not the peta-scale at work.

• ZFS is great at snapshots, replicating and not causing huge issues (or large penalties) along the way, but is a pain if you want to add a single disk
• BTRFS has a bad rep as not really production grade (IIRC mostly due to corruption bugs in their other than RAID1 modes and not having tools to properly maintain it), but I haven't worked with it.
• Ceph RBD with what ever you want on top, allows snapshots and replication, while allowing you to use whatever FS you like (though my experience is limited with it)
• CephFS is where you have way more experience than I have, as CephFS is only used within K8s volumes at work and there backup, expansion,... are all different

Backup Tools

There are a couple that come to my mind, but choosing tools is always difficult... Assuming we stay with file based backups (either from snapshots or the live FS)

• Restic (fairly efficient, way better than Borg in that department)
• Kopia (this is where quite some software stacks are moving to, mostly for efficiency)
• Borg (nice dedup, though slow due to it's rolling block based backup, which pretty much touches every file, single threaded, fully locks the backup repo)
• rdiff-backup (it's been a while, works somewhat similar to rsync, but does incremental forever backups)
• Proxmox Backup Server (shines on VMs with CBT, file based is ok, dedup and concurrency work very well at least in my setups)

There are better and worse ways in which backup applications traverse / search for changes in file systems, but in general file based backup is expensive *period* (virtual) disk backups usually deal with backups using some form of *change block tracking* (CBT, please don't google *CBT*, I've made that mistake once... use *change block tracking* as the search term) also called *dirty bit map* (remembering which blocks were changed), which is comparatively efficient, if data isn't modified much / only metadata is touched (and the file system stores metadata separately). This is implemented in different ways, though many of the solutions rely on some form of snapshots to achieve this. Snapshoting data no matter if on file level or a disk (the disk is also just a file, handled as if it was many small files for tracking) will create some performance penalties as it's either COW (copy on write) or COR (copy on read) unless CBT is used. Some CBT mechanisms work around the COW/COR issue by only ever storing the latest data and just noting, that it has changed (CBT) since the last backup, which allows them to be more efficient when it comes to the impact of changes (though the tracking and checking also takes some performance, but mostly CPU not IO). From a solution perspective, there is no *magic* solution to the issue, there are tradeoffs between IO, CPU, memory, bandwidth and data consistency (consistency is where snapshots and CBT shine). I've extensively used Borg for a while and must say, that I've started turning away from it due to the already mentioned issue with repo locking (as well as inefficient tracking). #### File Systems I cannot really comment on the "which file system to use" part from my work background, so what I'm commenting here is my ~20TB net home environment, not the peta-scale at work. * ZFS is great at snapshots, replicating and not causing huge issues (or large penalties) along the way, but is a pain if you want to add a single disk * BTRFS has a bad rep as not really production grade (IIRC mostly due to corruption bugs in their other than RAID1 modes and not having tools to properly maintain it), but I haven't worked with it. * Ceph RBD with what ever you want on top, allows snapshots and replication, while allowing you to use whatever FS you like (though my experience is limited with it) * CephFS is where you have way more experience than I have, as CephFS is only used within K8s volumes at work and there backup, expansion,... are all different #### Backup Tools There are a couple that come to my mind, but choosing tools is always difficult... Assuming we stay with file based backups (either from snapshots or the live FS) * Restic (fairly efficient, way better than Borg in that department) * Kopia (this is where quite some software stacks are moving to, mostly for efficiency) * Borg (nice dedup, though slow due to it's rolling block based backup, which pretty much touches every file, single threaded, fully locks the backup repo) * rdiff-backup (it's been a while, works somewhat similar to rsync, but does incremental forever backups) * Proxmox Backup Server (shines on VMs with CBT, file based is ok, dedup and concurrency work very well at least in my setups)

fnetX commented 2023年01月18日 18:46:58 +01:00

Note for investigation:

https://docs.ceph.com/en/latest/dev/cephfs-mirroring/

Note for investigation: https://docs.ceph.com/en/latest/dev/cephfs-mirroring/

fnetX commented 2023年01月30日 18:21:30 +01:00

A proof of concept backup for Git repositories is now running using rsync and retiring BorgBackup for repository backups. We'll keep using BorgBackup for large files (attachments, Git LFS, avatars etc) for now (I hope it performs acceptable there for now).

If the POC works out, I'm hoping we can implement a reusable solution that suits other large Forgejo instances, too.

A proof of concept backup for Git repositories is now running using rsync and retiring BorgBackup for repository backups. We'll keep using BorgBackup for large files (attachments, Git LFS, avatars etc) for now (I hope it performs acceptable there for now). If the POC works out, I'm hoping we can implement a reusable solution that suits other large Forgejo instances, too.

mkalus commented 2023年02月08日 13:07:51 +01:00

• ZFS is great at snapshots, replicating and not causing huge issues (or large penalties) along the way, but is a pain if you want to add a single disk

• Restic (fairly efficient, way better than Borg in that department)

I have been having good experiences with both zfs and restic on large data sets. Can definitely recommend both for your use case.

> * ZFS is great at snapshots, replicating and not causing huge issues (or large penalties) along the way, but is a pain if you want to add a single disk > * Restic (fairly efficient, way better than Borg in that department) I have been having good experiences with both zfs and restic on large data sets. Can definitely recommend both for your use case.

JavierVilarroig commented 2023年07月30日 17:36:44 +02:00

Have you taken a look at Proxmox Backup?

I have been using it at work for more than a year and I'm quite happy with the performance. In combination with Proxmox VE for LXC containers management is superb. But it's also working well on other environments.

It already has deduplication, incrementality, compression that helps reduce a lot the space and the time needed to run backups.

I can help with it if needed.

Have you taken a look at [Proxmox Backup](https://www.proxmox.com/en/proxmox-backup-server/overview)? I have been using it at work for more than a year and I'm quite happy with the performance. In combination with Proxmox VE for LXC containers management is superb. But it's also working well on other environments. It already has deduplication, incrementality, compression that helps reduce a lot the space and the time needed to run backups. I can help with it if needed.