forgejo-backup out of sync with database state? #7

mdk commented 2026年05月10日 14:37:18 +02:00

I'm wondering about the lost sync between repo backups and the database state.

I have not looked at the database content and I don't know to which point they need to be in sync to be able to cleanly restore a backup, have you?

Context: I own a small Forgejo instance (git.afpy.org) and was asking my self "hey, how does Codeberg handle backup at their scale?"). For the moment I stop forgejo, backup the DB run a forgejo dump and start forgejo, it takes two minutes at night, works only at my scale. But I feel nice about having the DB and the file backuped at the same moment.

I'm wondering about the lost sync between repo backups and the database state. I have **not** looked at the database content and I don't know to which point they need to be in sync to be able to cleanly restore a backup, have you? Context: I own a small Forgejo instance (git.afpy.org) and was asking my self "hey, how does Codeberg handle backup at their scale?"). For the moment I stop forgejo, backup the DB run a `forgejo dump` and start forgejo, it takes two minutes at night, works only at my scale. But I feel nice about having the DB and the file backuped at the same moment.

fnetX commented 2026年05月10日 15:02:45 +02:00

Yes, db and files are not in sync at this time. For most repos, this is not a big deal due to low activity. If we have to restore all of our repos for whatever reason, we'll have to run a few maintenance jobs to fix potential issues. There are some existing cron jobs to sync branches and tags from Git to database, for example.

I expect that there is potential breakage in pull requests, e.g. when a PR was created and either the Git backup or database is more recent, we'll likely run into issues (some broken PRs). We have to accept some compromises here due to the scale and keep the backups mostly for the convenience of a quick restore (= not requiring everyone to push back their work).

For migration tests on upcoming Forgejo versions, we take database + ceph snapshots at about the same time. It is not strict consistency, but at least very close to it. We could also investigate moving the backups to using this approach over the independent backups. But this requires effort to set this up, get it right and test it properly.

Yes, db and files are not in sync at this time. For most repos, this is not a big deal due to low activity. If we have to restore all of our repos for whatever reason, we'll have to run a few maintenance jobs to fix potential issues. There are some existing cron jobs to sync branches and tags from Git to database, for example. I expect that there is potential breakage in pull requests, e.g. when a PR was created and either the Git backup or database is more recent, we'll likely run into issues (some broken PRs). We have to accept some compromises here due to the scale and keep the backups mostly for the convenience of a quick restore (= not requiring everyone to push back their work). For migration tests on upcoming Forgejo versions, we take database + ceph snapshots at about the same time. It is not strict consistency, but at least very close to it. We could also investigate moving the backups to using this approach over the independent backups. But this requires effort to set this up, get it right and test it properly.

mdk commented 2026年05月10日 15:15:24 +02:00

Thanks for your answer!

Have a restore been tested? Are the "few maintenance jobs to fix potential issues" existing tools?

Thanks for your answer! Have a restore been tested? Are the "few maintenance jobs to fix potential issues" existing tools?

fnetX commented 2026年05月10日 15:22:35 +02:00

Have a restore been tested?

We have restored repos from the backups, yes. Mostly for things users or admins deleted inadvertently.

Are the "few maintenance jobs to fix potential issues" existing tools?

see

There are some existing cron jobs to sync branches and tags from Git to database, for example.

So yes, there is tooling for fixing issues. There are corruption issues that are impossible to repair, though (e.g. a database backup containing a PR reference but we don't have the Git data for this PR because it was not yet backed up).

> Have a restore been tested? We have restored repos from the backups, yes. Mostly for things users or admins deleted inadvertently. > Are the "few maintenance jobs to fix potential issues" existing tools? see > There are some existing cron jobs to sync branches and tags from Git to database, for example. So yes, there is tooling for fixing issues. There are corruption issues that are impossible to repair, though (e.g. a database backup containing a PR reference but we don't have the Git data for this PR because it was not yet backed up).

mdk commented 2026年05月10日 18:35:47 +02:00

OK, thank you for your time and answers! That's enlightening. While I'm here thanks for Codeberg!

OK, thank you for your time and answers! That's enlightening. While I'm here thanks for Codeberg!