Build infrastructure / container base images / image policy #60

gapodo commented 2022年12月06日 21:07:44 +01:00

I'm opening it here, since this spans multiple concerns and does not propperly fit the existing repos.

Intro / Overview

Thinking and deciding on a structure in which we maintain our container base images, helper images for pipelines, etc.

Quick note: I'm using OCI image and image instead of docker image as that's the vendor agnostic specification (an actual specification), which is also "guaranteed" portable to podman, crio, containerd and a lot of other container platforms (some based on containerd or crio, as well as others).

Goal

What I am trying to achive is a good compromise of reproducability (somewhat linked to #29), security, reliability and ease of maintenance, that is agreed upon, documented and will hopefully be followed.

Status quo

We are using OCI images in multiple places

• in our build pipelines
• as base for our OCI image releases

these Images are currently a mixture of

• directly pulled from docker.io
• built by us
  • from scratch
  • based on other images which are than again
    • built by us
    • taken from docker.io (and potentially other sources)

The images built by us, with exception of the automated release builds, are currently built by hand.

There is no procedure / task / schedule for mainenance on these images.

In pipelines we run a lot of "preparation" installs on our base images, which could just be maintained within a image we can than utilise.

We do not have fallback options if the image repo screws up layers (a bug sadly happening on Codeberg curtesy of Gitea and if we don't fix it downstream in the future forgejo).

Potential solutions / ideas

Organisational:

• Collect a team of people willing to help maintain and improve these images.
• Setup policies on how we handle version upgrades, tagging, security updates, etc. for these images.
• Define a policy on how we inform / update forgejo pipeline images and images used for our releases (images are defined in the forgejo/forgejo repo).
• Setup policies for retention / removal of images (think storage consumption as well as unfixed security issues).

Mixed:

• Add a new repo to forgejo (or create a organisation), with the concern of our base and infrastructure images, which contains the definitions (currently Dockerfiles) and pipelines used to generate them. (I am assuming we want to seperate it from the IaC repo)

Technical:

• Setup automation for at least building the images on change / manual trigger.
• Setup image scanning at least on build, to check for already known security issues.
• Potentially store exports of upstream images critical to our operation as export (in another repo, in releases or something) xgo comes to my mind as one of the most "irreplacable" ones (at least quickly)

Input welcome!

If you have any ideas (or opinions, I actually like those to), please share them.
Maintaining images, setting policies and keeping up with security is a wide field... and in some parts even big corporations can't keep up.

I'm opening it here, since this spans multiple concerns and does not propperly fit the existing repos. ## Intro / Overview Thinking and deciding on a structure in which we maintain our container base images, helper images for pipelines, etc. Quick note: I'm using *OCI image* and *image* instead of *docker image* as that's the vendor agnostic specification (an actual specification), which is also "guaranteed" portable to podman, crio, containerd and a lot of other container platforms (some based on containerd or crio, as well as others). ## Goal What I am trying to achive is a good compromise of reproducability (somewhat linked to #29), security, reliability and ease of maintenance, that is agreed upon, documented and will hopefully be followed. ## Status quo We are using OCI images in multiple places * in our build pipelines * as base for our OCI image releases these Images are currently a mixture of * directly pulled from docker.io * built by us * from scratch * based on other images which are than again * built by us * taken from docker.io (and potentially other sources) The images built by us, with exception of the automated release builds, are currently built by hand. There is no procedure / task / schedule for mainenance on these images. In pipelines we run a lot of "preparation" installs on our base images, which could just be maintained within a image we can than utilise. We do not have fallback options if the image repo screws up layers (a bug sadly happening on Codeberg curtesy of Gitea and if we don't fix it downstream in the future forgejo). ## Potential solutions / ideas ### Organisational: - [ ] Collect a team of people willing to help maintain and improve these images. - [ ] Setup policies on how we handle version upgrades, tagging, security updates, etc. for these images. - [ ] Define a policy on how we inform / update forgejo pipeline images and images used for our releases (images are defined in the forgejo/forgejo repo). - [ ] Setup policies for retention / removal of images (think storage consumption as well as unfixed security issues). ### Mixed: - [ ] Add a new repo to forgejo (or create a organisation), with the concern of our base and infrastructure images, which contains the definitions (currently Dockerfiles) and pipelines used to generate them. (I am assuming we want to seperate it from the IaC repo) ### Technical: - [ ] Setup automation for at least building the images on change / manual trigger. - [ ] Setup image scanning at least on build, to check for already known security issues. - [ ] Potentially store exports of upstream images critical to our operation as export (in another repo, in releases or something) xgo comes to my mind as one of the most "irreplacable" ones (at least quickly) ## Input welcome! If you have any ideas (or opinions, I actually like those to), please share them. Maintaining images, setting policies and keeping up with security is a wide field... and in some parts even big corporations can't keep up.

gapodo commented 2022年12月06日 21:09:35 +01:00

Forgot to add that this is also related to #58

Forgot to add that this is also related to #58

Ghost commented 2022年12月07日 00:18:58 +01:00

I think this all makes perfect sense and is essential to the long term maintenance and sanity of the CI/CD pipelines. Otherwise there is accumulated technical debt and undocumented builds that may or may not be reproducible all over the place.

I think this all makes perfect sense and is essential to the long term maintenance and sanity of the CI/CD pipelines. Otherwise there is accumulated technical debt and undocumented builds that may or may not be reproducible all over the place.

Ryuno-Ki commented 2022年12月07日 16:09:26 +01:00

For reference regarding Docker security:

https://snyk.io/learn/docker-security-scanning/

For reference regarding Docker security: https://snyk.io/learn/docker-security-scanning/

gapodo commented 2022年12月07日 16:50:43 +01:00

For reference regarding Docker security:

https://snyk.io/learn/docker-security-scanning/

Thanks, I was thinking of snyk, and/or clair as potential contenders (just from memory), we may want to look into this once we have the basics setup!

> For reference regarding Docker security: > > https://snyk.io/learn/docker-security-scanning/ Thanks, I was thinking of snyk, and/or clair as potential contenders (just from memory), we may want to look into this once we have the basics setup!

Ghost commented 2022年12月13日 13:54:32 +01:00

On the topic of version tagging, copy/pasting from an issue:

Container images are published at https://codeberg.org/forgejo/-/packages/container/forgejo/versions for stable versions. The "latest" version should point to the most recent stable version so that people can use it and expect upgrades to happen without requiring manual operation.

However this does not work for Forgejo because every major release is expected to require some kind of manual intervention and carefully reading the release notes before an upgrade is done.

For this reason tags such as v1.18 are set for every release and matches the latest point release v1.18.0-0, then v1.18.0-1 etc. The point releases can be upgraded without manual intervention (there has been exceptions with the first v1.16 point releases in Gitea but Forgejo can be careful to avoid that kind of predictible breakage).

On the topic of version tagging, copy/pasting from [an issue](https://codeberg.org/forgejo/forgejo/issues/73): Container images are published at https://codeberg.org/forgejo/-/packages/container/forgejo/versions for stable versions. The "latest" version should point to the most recent stable version so that people can use it and expect upgrades to happen without requiring manual operation. However this does not work for Forgejo because **every major release is expected to require some kind of manual intervention** and carefully reading the release notes before an upgrade is done. For this reason tags such as **v1.18** are set for every release and matches the latest point release **v1.18.0-0**, then **v1.18.0-1** etc. The point releases can be upgraded without manual intervention (there has been exceptions with the first **v1.16** point releases in Gitea but Forgejo can be careful to avoid that kind of predictible breakage).

gapodo commented 2022年12月13日 14:01:10 +01:00

I'd tend to upload a "dummy" image (echo "pleas use a tagged version" and exit 1 or similar) to latest, which just fails, as the whining is going to be huge when we do a major version jump and things just break on their own, because the users system auto pulls latest...

We may choose to release a "dev" container, that is the latest build of what ever is merged (for testing, similar to woodpecker-ci|s next tag), but I'd not make containers people use in prod just break (and latest would run exactly that risk).

I'd tend to upload a "dummy" image (echo "pleas use a tagged version" and exit 1 or similar) to latest, which just fails, as the whining is going to be huge when we do a major version jump and things just break on their own, because the users system auto pulls latest... We may choose to release a "dev" container, that is the latest build of what ever is merged (for testing, similar to woodpecker-ci|s next tag), but I'd not make containers people use in prod just break (and latest would run exactly that risk).

caesar commented 2022年12月13日 19:34:43 +01:00

However this does not work for Forgejo because every major release is expected to require some kind of manual intervention and carefully reading the release notes before an upgrade is done.

• Could you clarify what you mean by "major release"? Are we talking eg v1.18.0 (which I would call a "minor" release, though we don't use semver) or a potential v2.0.0?

• Is the problem here (automatic upgrades to major releases) because we don't use semver, or does Docker not respect semver anyway?
  In the latter case, presumably automatic breakage is normal in the Docker ecosystem if automatic updates are enabled - how do other projects handle this?

> However this does not work for Forgejo because every major release is expected to require some kind of manual intervention and carefully reading the release notes before an upgrade is done. - Could you clarify what you mean by "major release"? Are we talking eg `v1.18.0` (which I would call a "minor" release, though we don't use semver) or a potential `v2.0.0`? - Is the problem here (automatic upgrades to major releases) because we don't use semver, or does Docker not respect semver anyway? In the latter case, presumably automatic breakage is normal in the Docker ecosystem if automatic updates are enabled - how do other projects handle this?

Ghost commented 2022年12月13日 19:45:47 +01:00

Could you clarify what you mean by "major release"? Are we talking eg v1.18.0 (which I would call a "minor" release, though we don't use semver) or a potential v2.0.0?

I should have written release instead of major release to avoid confusion. I meant v1.16 => v1.17 => v1.18 etc.

Is the problem here (automatic upgrades to major releases) because we don't use semver, or does Docker not respect semver anyway?

The need for manual intervention is not specific to container images, it is because each Gitea release requires some kind of manual intervention. At the very minimum stopping the server, performing a full backup, upgrading, restarting and carefully verifying everything still works.

> Could you clarify what you mean by "major release"? Are we talking eg v1.18.0 (which I would call a "minor" release, though we don't use semver) or a potential v2.0.0? I should have written **release** instead of **major release** to avoid confusion. I meant v1.16 => v1.17 => v1.18 etc. > Is the problem here (automatic upgrades to major releases) because we don't use semver, or does Docker not respect semver anyway? The need for manual intervention is not specific to container images, it is because each Gitea release requires some kind of manual intervention. At the very minimum stopping the server, performing a full backup, upgrading, restarting and carefully verifying everything still works.

Ghost commented 2023年01月06日 17:07:17 +01:00

In the past week the release process underwent major changes and there seems to be a consensus that it is now both more secure and more robust than it ever was.

In addition the SemVer discussion branched to its own issue at forgejo/forgejo#207

Reason why I think this issue can be closed. If you feel differently, please re-open: no need to explain, just do that.

In the past week the release process underwent major changes and there seems to be a consensus that it is now both more secure and more robust than it ever was. In addition the SemVer discussion branched to its own issue at https://codeberg.org/forgejo/forgejo/issues/207 Reason why I think this issue can be closed. If you feel differently, please re-open: no need to explain, just do that.