Creation of archive.forgejo.org instance using s3-based object storage #474

crystal commented 2026年06月04日 19:52:06 +02:00

The devops team has recently been faced with a higher than normal number of disk cleanup operations due to the rapid growth of assets stored on code.forgejo.org as well as other things that are hosted by our k8s cluster. Codeberg has also recently expressed that they are not happy with the amount of storage and bandwidth the Forgejo project uses on their servers with uncompressed binary downloads and historical releases in a variety of different formats.

We do wish to hang on to all of these historical release artifacts for archival purposes, but they do not serve any useful purpose being fully duplicated across our high availability k8s cluster where storage is already at a premium. We intend to setup a new instance (archive.forgejo.org) which will store very old release artifacts in a cost-optimized s3 bucket, at which point we can delete them from code.forgejo.org and add the archive link as an external release asset. We can also setup a cron actions workflow to automate this process.

We would also like to start encouraging people to downloading release artifacts for recent versions from code.forgejo.org instead of Codeberg. We are considering also switching cfo to use s3-based object storage for this sort of thing, although in this case we will want to use a bucket with high availability and performance, and/or a finely-tuned caching webserver in front of it to ensure speedy release binary downloads for everyone.

Hopefully this will eventually allow us to offload most of the bulk data stored on our k8s cluster and ensure that limited storage capacity will not threaten to hinder the growth of the Forgejo project or it's ecosystem on code.forgejo.org.

The devops team has recently been faced with a higher than normal number of disk cleanup operations due to the rapid growth of assets stored on code.forgejo.org as well as other things that are hosted by our k8s cluster. Codeberg has also recently expressed that they are not happy with the amount of storage and bandwidth the Forgejo project uses on their servers with uncompressed binary downloads and historical releases in a variety of different formats. We do wish to hang on to all of these historical release artifacts for archival purposes, but they do not serve any useful purpose being fully duplicated across our high availability k8s cluster where storage is already at a premium. We intend to setup a new instance (`archive.forgejo.org`) which will store very old release artifacts in a cost-optimized s3 bucket, at which point we can delete them from code.forgejo.org and add the archive link as an external release asset. We can also setup a cron actions workflow to automate this process. We would also like to start encouraging people to downloading release artifacts for recent versions from code.forgejo.org instead of Codeberg. We are considering also switching cfo to use s3-based object storage for this sort of thing, although in this case we will want to use a bucket with high availability and performance, and/or a finely-tuned caching webserver in front of it to ensure speedy release binary downloads for everyone. Hopefully this will eventually allow us to offload most of the bulk data stored on our k8s cluster and ensure that limited storage capacity will not threaten to hinder the growth of the Forgejo project or it's ecosystem on code.forgejo.org.

cdotnow commented 2026年06月17日 20:30:32 +02:00

All ok, but why this has to be storage model based on s3? There are many more suited object storage models out there.

All ok, but why this has to be storage model based on `s3`? There are many more suited object storage models out there.

viceice commented 2026年06月30日 10:14:57 +02:00

@cdotnow wrote in #474 (comment):

All ok, but why this has to be storage model based on s3? There are many more suited object storage models out there.

it needs to be supported by forgejo 🙃

@cdotnow wrote in https://codeberg.org/forgejo/discussions/issues/474#issuecomment-17633810: > All ok, but why this has to be storage model based on `s3`? There are many more suited object storage models out there. it needs to be supported by forgejo 🙃

fnetX commented 2026年06月30日 19:29:06 +02:00

No objection, just for the record: I think that cleanup should mean "delete", and not moving stuff around all the time. If something is useful, it can stay where it is. If it is old, it should just be deleted.

Stopping uncompressed binaries and deleting the old stuff will immediately give you plenty of free storage, I suppose.

No objection, just for the record: I think that cleanup should mean "delete", and not moving stuff around all the time. If something is useful, it can stay where it is. If it is old, it should just be deleted. Stopping uncompressed binaries and deleting the old stuff will immediately give you plenty of free storage, I suppose.