Codeberg/Contributing

Fork 11

Code Issues 84 Wiki Activity

[Team] Backups #45

New issue

Open

opened 2024年01月30日 16:18:31 +01:00 by fnetX · 6 comments

fnetX added the

labels

2024年01月30日 16:18:31 +01:00

pyfisch commented

2024年02月14日 09:39:03 +01:00

Contributor

Copy link

Out of curiosity, how much much data from filesystems, git, s3 and databases needs to be backed up roughly?
Is hosted object storage (like Backblaze) an option to store the encrypted backups?
Should/must backups be append-only?

Regarding git backups: git bundle packs repositories into a single file, it allows for incremental backups as well.
If the storage consumed by forks is a concern, git has (with git alternates mechanism and namespaces) the option to share its object database between repositories. This is done by (that I know of) Gitlab and GitHub, but would likely require some engineering work in Codeberg and/or Forgejo.

* Out of curiosity, how much much data from filesystems, git, s3 and databases needs to be backed up roughly? * Is hosted object storage (like Backblaze) an option to store the encrypted backups? * Should/must backups be append-only? Regarding git backups: `git bundle` packs repositories into a single file, it allows for incremental backups as well. If the storage consumed by forks is a concern, git has (with git alternates mechanism and namespaces) the option to share its object database between repositories. This is done by (that I know of) Gitlab and GitHub, but would likely require some engineering work in Codeberg and/or Forgejo.

fnetX commented

2024年03月10日 23:31:24 +01:00

Author

Owner

Copy link

@pyfisch thank you for the comment.

Storage sizes:

database, gzip-compressed: 1.3 GB
repositories in backup (might lose some deduplcation): 3.1 TB
root filesystem backup: 2.0 TB in backup, about 520 GB on original disk (grows due to versioning)
attachments: 660 GB
packages: 770 GB
Git LFS: 500 GB

Hosted object storage can get pretty expensive over time. We moved to own hardware, because we estimated that the storage cost for hosted storage would grow too quickly. We also think that our current backup system saves a lot of energy and hardware.

It must not be possible to modify backups when the remote machine is compromised.

We are currently using https://codeberg.org/Codeberg-Infrastructure/forgejo-backup and it's working quite well. We already looked into Git alternates for the online storage, too, but it's very complex. At least GitHub is running a private soft-fork of Git anyway, so we suppose there are quite a lot of related changes. For Forgejo, we are considering to ditch forks anyway, see forgejo/discussions#131

@pyfisch thank you for the comment. Storage sizes: - database, gzip-compressed: 1.3 GB - repositories in backup (might lose some deduplcation): 3.1 TB - root filesystem backup: 2.0 TB in backup, about 520 GB on original disk (grows due to versioning) - attachments: 660 GB - packages: 770 GB - Git LFS: 500 GB Hosted object storage can get pretty expensive over time. We moved to own hardware, because we estimated that the storage cost for hosted storage would grow too quickly. We also think that our current backup system saves a lot of energy and hardware. It must not be possible to modify backups when the remote machine is compromised. We are currently using https://codeberg.org/Codeberg-Infrastructure/forgejo-backup and it's working quite well. We already looked into Git alternates for the online storage, too, but it's very complex. At least GitHub is running a private soft-fork of Git anyway, so we suppose there are quite a lot of related changes. For Forgejo, we are considering to ditch forks anyway, see https://codeberg.org/forgejo/discussions/issues/131

pyfisch commented

2024年03月11日 20:37:54 +01:00

Contributor

Copy link

That is smaller than I expected, less than 10 TB in total.

Hosted object storage can get pretty expensive over time. We moved to own hardware, because we estimated that the storage cost for hosted storage would grow too quickly.

Backblaze B2 is 6$/TB/month, so 60€/month at most. https://www.backblaze.com/cloud-storage/pricing
I am not sure a dedicated server or VPS with RAID is significantly cheaper but is more likely to fail.

In the other discussion (linked below) Ceph mirroring and filesystem snapshots are discussed. Do you want to use that or is file based backup with tools like Borg or restic sufficient?

Related discussion:

That is smaller than I expected, less than 10 TB in total. > Hosted object storage can get pretty expensive over time. We moved to own hardware, because we estimated that the storage cost for hosted storage would grow too quickly. Backblaze B2 is 6$/TB/month, so 60€/month at most. https://www.backblaze.com/cloud-storage/pricing I am not sure a dedicated server or VPS with RAID is significantly cheaper but is more likely to fail. In the other discussion (linked below) Ceph mirroring and filesystem snapshots are discussed. Do you want to use that or is file based backup with tools like Borg or restic sufficient? Related discussion: * Codeberg-Infrastructure/planning#4 * Codeberg-Infrastructure/techstack-support#5

JavierVilarroig commented

2024年03月11日 21:40:49 +01:00

Copy link

My experience with Proxmox Backup has been very good.

Instead of going for expensive RAID hardware configuration, I have two servers in different locations and periodic jobs for data replication.

I have a third server in my working laptop where part of the backups are replicated for long time/permanent storage.

It has support for off line tape storage.

It excels at Proxmox VE backups. But I also use it as a CLI backup tool with very good results.

It has a good API that allows good integration with standard tools. I use the API to interact with PBE from a Jenkins CI system.

PBS is an evolving backup system that I think we can rely on.

I can help building and maintaining a PBS setup if needed.

My experience with Proxmox Backup has been very good. Instead of going for expensive RAID hardware configuration, I have two servers in different locations and periodic jobs for data replication. I have a third server in my working laptop where part of the backups are replicated for long time/permanent storage. It has support for off line tape storage. It excels at Proxmox VE backups. But I also use it as a CLI backup tool with very good results. It has a good API that allows good integration with standard tools. I use the API to interact with PBE from a Jenkins CI system. PBS is an evolving backup system that I think we can rely on. I can help building and maintaining a PBS setup if needed.

fnetX commented

2024年03月14日 22:19:59 +01:00

Author

Owner

Copy link

We don't use Proxmox, but looking at good backup solutions is usually worth a try.

The backup of repositories currently works rather well.

Backup of LFS and Packages is probably of the lowest priority, because the content is possible to restore. Backups here are for the convenience of our users and to allow for a quick restoration after a disaster.

Highest priority have files that are only stored on Codeberg. Our MariaDB database is a crucial part of collaboration, and I think that most attachments (in issue trackers etc) should be backed up regularly.

What also deserves some work is the host OS including containers. I once wanted to restore a file from a backup and it really took me a while to mount and restore that. My idea would be to use BTRFS filesystem snapshots per LXC container so that admins can access them on their own. And maybe use something dedicated to this in order to move a daily snapshot to an offsite location.

Once this is done, I think we should have a look at encrypted offsite backups and improve the current situation. Some Codeberg members also offered to donate some storage, and I think it's worth a consideration, because it is probably more efficient than cloud providers, and maybe faster to restore, because many sources can feed at once. The problem is that many encrypted offsite backup require decryption keys to run housekeeping, and running housekeeping via remote requires quite some traffic and resources. Is there a tool that can run automatic prune jobs on a remote machine without trusting it? (For a friend, I am using Syncthing as a backup and it does exactly this: It feeds encrypted data to my machine, my machine does staggered versioning and prunes old history. Syncthing is of course only abused in this scenario, it is not made for backups at scale).

We don't use Proxmox, but looking at good backup solutions is usually worth a try. The backup of repositories currently works rather well. Backup of LFS and Packages is probably of the lowest priority, because the content is possible to restore. Backups here are for the convenience of our users and to allow for a quick restoration after a disaster. Highest priority have files that are only stored on Codeberg. Our MariaDB database is a crucial part of collaboration, and I think that most attachments (in issue trackers etc) should be backed up regularly. What also deserves some work is the host OS including containers. I once wanted to restore a file from a backup and it really took me a while to mount and restore that. My idea would be to use BTRFS filesystem snapshots *per LXC container* so that admins can access them on their own. And maybe use something dedicated to this in order to move a daily snapshot to an offsite location. Once this is done, I think we should have a look at encrypted offsite backups and improve the current situation. Some Codeberg members also offered to donate some storage, and I think it's worth a consideration, because it is probably more efficient than cloud providers, and maybe faster to restore, because many sources can feed at once. The problem is that many encrypted offsite backup require decryption keys to run housekeeping, and running housekeeping via remote requires quite some traffic and resources. Is there a tool that can run automatic prune jobs on a remote machine without trusting it? (For a friend, I am using Syncthing as a backup and it does exactly this: It feeds encrypted data to my machine, my machine does staggered versioning and prunes old history. Syncthing is of course only abused in this scenario, it is not made for backups at scale).

AfroThundr commented

2024年04月15日 00:03:28 +02:00

Copy link

Backblaze B2 is 6$/TB/month, so 60€/month at most. https://www.backblaze.com/cloud-storage/pricing
I am not sure a dedicated server or VPS with RAID is significantly cheaper but is more likely to fail.

Another cheap cloud alternative to this would be Wasabi. They're based on Amazon S3 but with much cheaper pricing since they can scale enough to make it work. They have similar pricing to Backblaze. I have ~8TB of Borg backup repositories stored with them and haven't passed 60ドル/mo yet. Yet another solution to consider is Rsync.net with similar pricing. There's plenty of options that focus on this type of use case and make it more affordable (especially taking into account availability and reliability vs self-hosted hardware).

What also deserves some work is the host OS including containers. I once wanted to restore a file from a backup and it really took me a while to mount and restore that. My idea would be to use BTRFS filesystem snapshots per LXC container so that admins can access them on their own. And maybe use something dedicated to this in order to move a daily snapshot to an offsite location.

I haven't played much with BTRFS since it's still the new kid on the block in this space, but I use ZFS for my local storage pools. I get the same deduplication and compression, plus the ability to zfs send snapshots over ssh to a remote machine for backups. I know BTRFS has similar capabilities as far as dedup/compression and subvolumes, etc, and they can apparently also do send/receive to remote machines. That could be useful for local backups at least, and possibly for backups to remotes that don't have to worry much about bandwidth (though both filesystems support incremental send to mitigate this).

Once this is done, I think we should have a look at encrypted offsite backups and improve the current situation. Some Codeberg members also offered to donate some storage, and I think it's worth a consideration, because it is probably more efficient than cloud providers, and maybe faster to restore, because many sources can feed at once.

If you're fine with the relatively mediocre "two nines" reliability of a residential internet connection, I too can offer storage space on one of my storage nodes. A 10TB ZFS subvolume to play with would be a decent start yes? I'm also hosting mirrors of lots of other stuff on this node and have plenty of space to spare should it be needed. Assuming your backups are encrypted you don't have to fully trust the remote that's hosting your data. A nice property to have (and why I'm ok with using cloud providers for my off-site backups in the first place).

The problem is that many encrypted offsite backup require decryption keys to run housekeeping, and running housekeeping via remote requires quite some traffic and resources. Is there a tool that can run automatic prune jobs on a remote machine without trusting it? (For a friend, I am using Syncthing as a backup and it does exactly this: It feeds encrypted data to my machine, my machine does staggered versioning and prunes old history. Syncthing is of course only abused in this scenario, it is not made for backups at scale).

This is where I will plug Borg backup again. You can do maintenance like pruning old archives and compacting the repository to reduce space usage from pruned backups without needing a key to the repository. I use this for all my VMs and nodes that have data I care about retaining. They all push backups to this node, with the repositories in "append only" mode in the event of a compromise of the remote node (they can't actually "delete" a backup, only mark it as deleted, which can be undone). Then on a monthly basis, the storage node will disable remote access, take the repositories out of append only mode, and runs a verify and compact job on each one. Then the reverse happens, the repositories are made append only again, access is restored, and backups can resume.

Borg did have some performance bottlenecks with certain large datasets, but much if this was mitigated by breaking up what needed to be archived and where. Certain datasets that didn't need to be put together in one archive got separate archives and backup jobs, allowing them to run independently/concurrently without affecting each other. Larger single datasets were backed up in smaller chunks (ex: per top level directory) then a final run over the whole set (usually producing no additional data usage). Combine that with proper IO scheduling and network QoS and you barely notice the jobs running.

To get the cloud storage remote backups, I use rclone to then sync the local repositories to Wasabi, but any other backend supported by rclone (a lot) would work here. The combined workflow results in deduplicated and compressed, encrypted incremental (and client immutable) backups, with off-site mirrors of those backups. And the only ones with the keys to those backups are the originating clients (and me in case I need to do a full recovery).

> Backblaze B2 is 6$/TB/month, so 60€/month at most. https://www.backblaze.com/cloud-storage/pricing I am not sure a dedicated server or VPS with RAID is significantly cheaper but is more likely to fail. Another cheap cloud alternative to this would be [Wasabi](https://wasabi.com/cloud-storage-pricing/#three-info). They're based on Amazon S3 but with much cheaper pricing since they can scale enough to make it work. They have similar pricing to Backblaze. I have ~8TB of Borg backup repositories stored with them and haven't passed 60ドル/mo yet. Yet another solution to consider is [Rsync.net](https://rsync.net/pricing.html) with similar pricing. There's plenty of options that focus on this type of use case and make it more affordable (especially taking into account availability and reliability vs self-hosted hardware). > What also deserves some work is the host OS including containers. I once wanted to restore a file from a backup and it really took me a while to mount and restore that. My idea would be to use BTRFS filesystem snapshots *per LXC container* so that admins can access them on their own. And maybe use something dedicated to this in order to move a daily snapshot to an offsite location. I haven't played much with BTRFS since it's still the new kid on the block in this space, but I use ZFS for my local storage pools. I get the same deduplication and compression, plus the ability to `zfs send` snapshots over ssh to a remote machine for backups. I know BTRFS has similar capabilities as far as dedup/compression and subvolumes, etc, and they can apparently also do send/receive to remote machines. That could be useful for local backups at least, and possibly for backups to remotes that don't have to worry much about bandwidth (though both filesystems support incremental send to mitigate this). > Once this is done, I think we should have a look at encrypted offsite backups and improve the current situation. Some Codeberg members also offered to donate some storage, and I think it's worth a consideration, because it is probably more efficient than cloud providers, and maybe faster to restore, because many sources can feed at once. If you're fine with the relatively mediocre "two nines" reliability of a residential internet connection, I too can offer storage space on one of my storage nodes. A 10TB ZFS subvolume to play with would be a decent start yes? I'm also hosting mirrors of lots of other stuff on this node and have plenty of space to spare should it be needed. Assuming your backups are encrypted you don't have to fully trust the remote that's hosting your data. A nice property to have (and why I'm ok with using cloud providers for my off-site backups in the first place). > The problem is that many encrypted offsite backup require decryption keys to run housekeeping, and running housekeeping via remote requires quite some traffic and resources. Is there a tool that can run automatic prune jobs on a remote machine without trusting it? (For a friend, I am using Syncthing as a backup and it does exactly this: It feeds encrypted data to my machine, my machine does staggered versioning and prunes old history. Syncthing is of course only abused in this scenario, it is not made for backups at scale). This is where I will plug Borg backup again. You can do maintenance like pruning old archives and compacting the repository to reduce space usage from pruned backups without needing a key to the repository. I use this for all my VMs and nodes that have data I care about retaining. They all push backups to this node, with the repositories in "append only" mode in the event of a compromise of the remote node (they can't actually "delete" a backup, only mark it as deleted, which can be undone). Then on a monthly basis, the storage node will disable remote access, take the repositories out of append only mode, and runs a verify and compact job on each one. Then the reverse happens, the repositories are made append only again, access is restored, and backups can resume. Borg did have some performance bottlenecks with certain large datasets, but much if this was mitigated by breaking up what needed to be archived and where. Certain datasets that didn't need to be put together in one archive got separate archives and backup jobs, allowing them to run independently/concurrently without affecting each other. Larger single datasets were backed up in smaller chunks (ex: per top level directory) then a final run over the whole set (usually producing no additional data usage). Combine that with proper IO scheduling and network QoS and you barely notice the jobs running. To get the cloud storage remote backups, I use `rclone` to then sync the local repositories to Wasabi, but any other backend supported by rclone ([a lot](https://rclone.org/overview/)) would work here. The combined workflow results in deduplicated and compressed, encrypted incremental (and client immutable) backups, with off-site mirrors of those backups. And the only ones with the keys to those backups are the originating clients (and me in case I need to do a full recovery).

No Branch/Tag specified

Branches Tags

wiki-notice

main

Labels

Clear labels

Invalid

Off-Topic

This issue concerns a matter that is not relevant to this repository.

Invalid

Out Of Scope

This issue is beyond the scope of this repository.