So, I'm basically planning to build a CI which is integrated with Gitea and can handle all limitations and requirements of Codeberg. This is the proposed concept, I'd be glad to get a lot of feedback!
Requirements
- Securely build and deploy every commit, tag and pull request
- Use as little resources per build as possible
- Be as easy to use as possible, with the least possible effort for most repositories
- Show a progress report of each build and make artifacts available for download
- Automatically deploy & allocate servers on Hetzner
- Limit resource consumption by each user
- Limit monthly overall budget
- Show build status and possibly progress and artifact links directly in Gitea
- Possibly support for multi-arch builds
- Probably a lot more I can't currently think of
The plan
- Use Docker to build stuff and isolate builds from each other
- Use annotated Dockerfiles to specify all build instructions
→ many repositories already have a Dockerfile, so they would build automatically. If that's not wanted for a repository, a.cifilewould override theDockerfile; an empty.cifilewould basically disable the CI for this repository. - Keep the Docker Cache between builds, so new builds are faster
- Show the build progress as a graph derived from the annotations in the Dockerfile
- Use server-side receive hooks to trigger builds, and
POST /repos/{owner}/{repo}/statuses/{sha}to set the build status of a commit
Annotations could look like this:
FROMgolang:1ASbuild# define and run a subtask (if the "{" is left away, the file "setup" would be pasted here):# @include setup {COPY . /appWORKDIR/app# @}# run two tasks simultaneously, then wait for them to complete:# @required build {RUN go build -o exampleapp .# @}# @optional test {RUN go test# @}# any normal docker command or annotation except for "@optional" and "@required" would also wait for completion# @complete# deploy the binary as a download and push it to the codeberg.org Docker registry:# @required deploy {# @artifact /app/exampleapp # make artifact available for download# @if status != WARNING { # if the test succeeded...# # ...deploy as a docker image# @push codeberg.org/momar/myexampleapp:latest# @if tag != "" { # if the current commit has a tag attached...# # ...deploy as a tagged docker image# @push codeberg.org/momar/myexampleapp:${tag}# @}# @}# @}The resulting graph would then be that one:
Plugins
We could link to other Dockerfile templates (e.g. using pongo2) to run specific tasks:
# @do codeberg.org/momar/ci-plugins/create-release {# # this is the YAML plugin configuration# title: ${version}# tag: ${tag}# content: ${commit.message}# files:# build: /app/exampleapp # from the step "build", include the file "/app/exampleapp"# @if tag != version {# isPrerelease: true# @}# @}Finances
I'd suggest to limit by CPU cores, memory and parallel builds, an example based on the current membership fees, with Hetzner CX21 build servers, could look like this:
- each user and organisation (with more than 1 member) gets its own queue, resources are allocated within that queue
- the owner of an organisation with the biggest plan determines the plan of the organisation
- non-members get 0.5 cores with 1GB of RAM, and can only have one build running at a time
- members who pay 24€/year get 2 cores with 4GB of RAM, and can have 2 builds running at the same time (which then share those resources)
- for each additional 12€/year, they get an additional core, an additional 2GB of RAM, and an additional simultaneous build
- that way, members get 1 core for 1€ (that's excluding any other costs) - assuming that the average user is running 72 hours worth of builds in 30 days, that means that 40 paying users could fit on a single CX21 server for <6€ - with 24 paying users per server (so, 25% of the income goes towards the CI servers), there would be space for 64 non-paying users.
- the Docker cache could be stored on a Hetzner Storage Box, so limiting each repo to 5GB cache (and allowing exceptions) should be totally fine.
I'm not sure about how wrong my calculations are, and have no overview about our general finances, but it seems like it would be possible to support a CI system like that.
So, I'm basically planning to build a CI which is integrated with Gitea and can handle all limitations and requirements of Codeberg. This is the proposed concept, I'd be glad to get a lot of feedback!
## Requirements
- Securely build and deploy every commit, tag and pull request
- Use as little resources per build as possible
- Be as easy to use as possible, with the least possible effort for most repositories
- Show a progress report of each build and make artifacts available for download
- Automatically deploy & allocate servers on Hetzner
- Limit resource consumption by each user
- Limit monthly overall budget
- Show build status and possibly progress and artifact links directly in Gitea
- Possibly support for multi-arch builds
- Probably a lot more I can't currently think of
## The plan
- Use Docker to build stuff and isolate builds from each other
- Use annotated Dockerfiles to specify all build instructions
→ many repositories already have a Dockerfile, so they would build automatically. If that's not wanted for a repository, a `.cifile` would override the `Dockerfile`; an empty `.cifile` would basically disable the CI for this repository.
- Keep the Docker Cache between builds, so new builds are faster
- Show the build progress as a graph derived from the annotations in the Dockerfile
- Use server-side receive hooks to trigger builds, and `POST /repos/{owner}/{repo}/statuses/{sha}` to set the build status of a commit
Annotations could look like this:
```Dockerfile
FROM golang:1 AS build
# define and run a subtask (if the "{" is left away, the file "setup" would be pasted here):
# @include setup {
COPY . /app
WORKDIR /app
# @}
# run two tasks simultaneously, then wait for them to complete:
# @required build {
RUN go build -o exampleapp .
# @}
# @optional test {
RUN go test
# @}
# any normal docker command or annotation except for "@optional" and "@required" would also wait for completion
# @complete
# deploy the binary as a download and push it to the codeberg.org Docker registry:
# @required deploy {
# @artifact /app/exampleapp # make artifact available for download
# @if status != WARNING { # if the test succeeded...
# # ...deploy as a docker image
# @push codeberg.org/momar/myexampleapp:latest
# @if tag != "" { # if the current commit has a tag attached...
# # ...deploy as a tagged docker image
# @push codeberg.org/momar/myexampleapp:${tag}
# @}
# @}
# @}
```
The resulting graph would then be that one:
### Plugins
We could link to other Dockerfile templates (e.g. using [pongo2](https://github.com/flosch/pongo2)) to run specific tasks:
```Dockerfile
# @do codeberg.org/momar/ci-plugins/create-release {
# # this is the YAML plugin configuration
# title: ${version}
# tag: ${tag}
# content: ${commit.message}
# files:
# build: /app/exampleapp # from the step "build", include the file "/app/exampleapp"
# @if tag != version {
# isPrerelease: true
# @}
# @}
```
## Finances
I'd suggest to limit by CPU cores, memory and parallel builds, an example based on the current membership fees, with Hetzner CX21 build servers, could look like this:
- each user and organisation (with more than 1 member) gets its own queue, resources are allocated within that queue
- the owner of an organisation with the biggest plan determines the plan of the organisation
- non-members get 0.5 cores with 1GB of RAM, and can only have one build running at a time
- members who pay 24€/year get 2 cores with 4GB of RAM, and can have 2 builds running at the same time (which then share those resources)
- for each additional 12€/year, they get an additional core, an additional 2GB of RAM, and an additional simultaneous build
- that way, members get 1 core for 1€ (that's excluding any other costs) - assuming that the average user is running 72 hours worth of builds in 30 days, that means that 40 paying users could fit on a single CX21 server for <6€ - with 24 paying users per server (so, 25% of the income goes towards the CI servers), there would be space for 64 non-paying users.
- the Docker cache could be stored on a Hetzner Storage Box, so limiting each repo to 5GB cache (and allowing exceptions) should be totally fine.
I'm not sure about how wrong my calculations are, and have no overview about our general finances, but it seems like it would be possible to support a CI system like that.
152 KiB