Monthly Report for July 2025 #609

Beowulf commented 2025年07月05日 16:02:21 +02:00

Please collect snippets and points of interest for the monthly report here.

Please collect snippets and points of interest for the monthly report here.

mahlzahn commented 2025年07月05日 16:25:46 +02:00

Some notes/ideas already here:

• website: documentation has now a version selector forgejo/website#606 and sidebar forgejo/website#614
• ongoing discussion about contributor attribution forgejo/docs#1275
• ongoing discussion about usage of AI in Forgejo development forgejo/discussions#366
• talk by @jerger, about federation if happened, as advertised in June report
• releases: v12.0, EOL of v7.0 LTS. Maybe last arm/v6 version? forgejo/discussions#364
• sustainability: end of old NLnet grant: implemented some of the steps. 2022年12月01日-nlnet, maybe a recap, what was founded and made it into Forgejo.

Some notes/ideas already here: - website: documentation has now a version selector forgejo/website#606 and sidebar forgejo/website#614 - ongoing discussion about contributor attribution forgejo/docs#1275 - ongoing discussion about usage of AI in Forgejo development forgejo/discussions#366 - talk by @jerger, about federation if happened, as advertised in June report - releases: v12.0, EOL of v7.0 LTS. Maybe last arm/v6 version? forgejo/discussions#364 - sustainability: end of old NLnet grant: implemented some of the steps. [2022年12月01日-nlnet](https://codeberg.org/forgejo/sustainability/src/branch/main/2022-12-01-nlnet), maybe a recap, what was founded and made it into Forgejo.

earl-warren commented 2025年07月21日 12:20:06 +02:00

Forgejo releases

Forgejo v12.0.0 was published on time, 17 July. A regression involving Minio was caught and fixed just in time, less than 24h before the release. Another regression was found and fixed after the release was published. Workarounds were added to the release notes and it was decided to not publish a patch release right away.

A week later Forgejo v12.0.1 was published and reverts a breaking change that should have been prepared more carefully. An effort started and will hopefully allow to re-introduce it.

Tooling

The release process is not entirely automated yet but the release manager aims to do that eventually. The checklist of manual tasks was clarified. With a major release every three months, it takes time for ambiguities to surface.

The release notes assistant that collects and compiles release notes was the main pain point for this release because it took a lot of time to complete (about two hours). It was created a year ago with a focus on correctness rather than efficiency and it got to a point where it became a problem, holding back the release during hours for a process that should not take more than a few minutes. It was made 20 times faster and now completes within two minutes. See also the rationale for not using the most commonly known alternative.

The toolchain used to build Forgejo and Forgejo runner rely on the same action which builds the binary in a container for each supported architecture. This ensures the same binary is used standalone and in the container. It is specialized to Forgejo and is unlikely to be used for any other purpose.

However it relies on the forgejo-release action which has received a few contributions over the past year and is apparently used by other projects although it was first designed to be specific to Forgejo. It gained support for efficiently using the release-notes-assistant. The Forgejo runner v8.0.1 began using it. It won't be necessary to manually manage the inventory of pull requests. Only breaking changes and how to deal with them will have to be explained.

## Forgejo releases [Forgejo v12.0.0](https://forgejo.org/2025-07-release-v12-0/) was published on time, 17 July. A regression involving Minio was [caught and fixed](https://codeberg.org/forgejo/forgejo/issues/8529) just in time, less than 24h before the release. Another regression was found [and fixed](https://codeberg.org/forgejo/forgejo/pulls/8550) after the release was published. Workarounds were [added to the release notes](https://codeberg.org/forgejo/forgejo/milestone/12836) and it was decided to not publish a patch release right away. A week later [Forgejo v12.0.1](https://codeberg.org/forgejo/forgejo/milestone/22237) was published and reverts a breaking change that should have been prepared more carefully. An [effort started](https://codeberg.org/forgejo/forgejo/issues/8634) and will hopefully allow to re-introduce it. ### Tooling The release process is not entirely automated yet but [the release manager](https://code.forgejo.org/forgejo/release-manager/commits/branch/main) aims to do that eventually. The [checklist of manual tasks](https://code.forgejo.org/forgejo/release-manager/commit/c3d2c05f5b314fb86000a3e227e7dfc56e6516a7) was clarified. With a major release every three months, it takes time for ambiguities to surface. The [release notes assistant](https://code.forgejo.org/forgejo/release-notes-assistant) that collects and compiles release notes was the main pain point for this release because it took a lot of time to complete (about two hours). It was created a year ago with a focus on correctness rather than efficiency and it got to a point where it became a problem, holding back the release during hours for a process that should not take more than a few minutes. It was [made 20 times faster](https://code.forgejo.org/forgejo/release-notes-assistant/pulls/89/files) and now completes within two minutes. See also [the rationale for not using the most commonly known alternative](https://code.forgejo.org/forgejo/release-notes-assistant#alternatives). The toolchain used to build Forgejo and Forgejo runner rely on the [same action](https://code.forgejo.org/forgejo/forgejo-build-publish) which builds the binary in a container for each supported architecture. This ensures the same binary is used standalone and in the container. It is specialized to Forgejo and is unlikely to be used for any other purpose. However it relies on the [forgejo-release](https://code.forgejo.org/actions/forgejo-release) action which has received a few contributions over the past year and is apparently used by other projects although it was first designed to be specific to Forgejo. It [gained support for efficiently using the release-notes-assistant](https://code.forgejo.org/actions/forgejo-release/pulls/69/files). The [Forgejo runner v8.0.1](https://code.forgejo.org/forgejo/runner/releases/tag/v8.0.1) began using it. It won't be necessary to manually manage the inventory of pull requests. Only breaking changes and how to deal with them will have to be explained.

earl-warren commented 2025年07月21日 12:41:05 +02:00

Forgejo infrastructure

Disaster recover exercise

The Forgejo infrastructure migrated end of 2024 to a k8s cluster that is self hosted on three bare metal machines running only Free Software in Germany and Finland. It is designed to recover from a datacenter going down in either country without data loss. That is the theory and it was tested when the cluster was first setup. But it is good to practice that disaster recovery scenario from time to time and it happened 24 July 2025.

The detailed task list was published for archive. In a nutshell it went as well as can be expected, except for one detail: NFS backed Forgejo instances can be extremely slow when it involves going back and forth between German and Finland (a git clone can take 5 minutes instead of 15 seconds).

Network outage

Coincidentally on 21 July, the day the disaster recover exercise was announced, a network outage impacted the Forgejo Actions runners servicing Codeberg between midnight and 4am UTC. The bare metal machines hosted at Hetzner in Finland could no longer reach Codeberg while those hosted in Germany using the same exit point (core5.fra.hetzner.com) had no issues. A support request to Hetzner less than one hour after the outage began but it did not provide clarity on the problem and it apparently fixed itself. Or it was fixed and there was no information about it.

With that exception the Forgejo infrastructure required little to no attention from the devops team. A welcome pause compared to the work required to mitigate and understand the excessive crawling of the previous months.

Hardware costs

The cost of the Forgejo hardware infrastructure was broken down and the rationale for their existence explained. So far it has been paid for informally by individuals and a more sustainable model was discussed. It was proposed that the costs are covered by Codeberg and the necessary administrative steps were taken. The following months will tell if this transition works as well as can be expected.

## Forgejo infrastructure ### Disaster recover exercise The Forgejo infrastructure migrated end of 2024 to a [k8s cluster](https://codeberg.org/forgejo/k8s-cluster/) that is self hosted on three bare metal machines running only Free Software in Germany and Finland. It is designed to recover from a datacenter going down in either country without data loss. That is the theory and it was tested when the cluster was first setup. But it is good to practice that disaster recovery scenario from time to time and it happened [24 July 2025](https://codeberg.org/forgejo/discussions/issues/374). The [detailed task list was published](https://codeberg.org/forgejo/k8s-cluster/src/commit/f70ba83508af3f19d0caebf61b6a25d15eb89dbe/docs/k8s-2025年07月24日-disaster-recovery-exercise.md) for archive. In a nutshell it went as well as can be expected, except for one detail: NFS backed Forgejo instances can be extremely slow when it involves going back and forth between German and Finland (a git clone can take 5 minutes instead of 15 seconds). ### Network outage Coincidentally on 21 July, the day the disaster recover exercise was announced, a network outage impacted the Forgejo Actions runners servicing Codeberg between midnight and 4am UTC. The bare metal machines hosted at Hetzner in Finland could no longer reach Codeberg while those hosted in Germany using the same exit point (`core5.fra.hetzner.com`) had no issues. A support request to Hetzner less than one hour after the outage began but it did not provide clarity on the problem and it apparently fixed itself. Or it was fixed and there was no information about it. With that exception the Forgejo infrastructure required little to no attention from the devops team. A welcome pause compared to the work required to mitigate and understand [the excessive crawling](https://codeberg.org/forgejo/discussions/issues/320) of the previous months. ### Hardware costs The [cost of the Forgejo hardware infrastructure](https://codeberg.org/forgejo/sustainability/issues/104) was broken down and the rationale for their existence explained. So far it has been [paid for informally](https://codeberg.org/forgejo/sustainability/src/branch/main/README.md#2025) by individuals and a more sustainable model was discussed. It was proposed that the [costs are covered by Codeberg](https://codeberg.org/forgejo/governance/issues/303) and the necessary administrative steps were taken. The following months will tell if this transition works as well as can be expected.

earl-warren commented 2025年07月21日 13:17:43 +02:00

Forgejo Actions

The Forgejo runner v6.4.0 was published and unfortunately contained a serious regression. Once it was fixed Forgejo runner v7.0.0 was published and it was recommended to not use the previous version.

A few weeks later the longest standing security bug of the runner was fixed (i.e. redacting multiline secrets) and was published in the Forgejo runner v8.0.0 release.

This version also validates actions and workflows against a schema. It is not optional although it is a significant breaking change because:

• A workflow or action with random content may lead to unexpected behavior that is very difficult to figure out and debug
• When something works by chance but is not documented or supported, users will start to depend on it and it will have to be supported or migrated in a backward compatible way. This is can be very difficult once the runner is stable.
• Errors that are only detected at runtime by an incorrect behavior can be very difficult to relate to the root cause when it is an incorrect syntactic error in the workflow. It is much less difficult to figure out a syntactic error that is displayed in the Forgejo UI or in the output of a job that fails to run because of them.

No more GITHUB in variable names

Starting with v7.0.0, the runner allows to write new workflows without ever using github in the name of contexts or variables. It is mostly cosmetic, reason why it took years to land. The previous names that include github are preserved for backward compatibility and to facilitate the re-use of actions that were originally developed exclusively for GitHub. Shortly after this release was deployed in the Forgejo infrastructure, a series of pull requests was merged to take advantage of those nicer looking names.

Bugs and features reorganization

All issues and pull requests for ACT and the runner were scrubbed and triaged. Bug reports now are grouped in the Forgejo runner issue tracker and feature requests were moved to a dedicated space. On that occasion a new workflow was discussed and implemented. In a nutshell it tries to find balance between the expectations of someone filing a new feature request and the need for first hand testimonies to guide the implementation.

To not re-invent the wheel, an exhaustive inventory of the commits found in the Nektos ACT and act_runner repositories was done and compared to the Forgejo runner implementation. It had to go back one year but there has not been a lot of activity: it did not take too long and the commits that were discovered will save valuable development time.

Design and User Research

A design and user research effort (1 and 2) began to figure out how Forgejo Actions CI results should be followed. It will help figure out the next steps now that Forgejo v12.0 has the building blocks allowing it to send mail notifications when a workflow fails.

## Forgejo Actions The [Forgejo runner v6.4.0](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#6-4-0) was published and unfortunately contained a serious regression. Once it was fixed [Forgejo runner v7.0.0](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#7-0-0) was published and it was recommended to not use the previous version. A few weeks later the longest standing security bug of the runner was fixed (i.e. [redacting multiline secrets](https://code.forgejo.org/forgejo/runner/pulls/661)) and was published in the [Forgejo runner v8.0.0](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#8-0-0) release. This version also [validates actions and workflows against a schema](https://code.forgejo.org/forgejo/runner/src/branch/main/RELEASE-NOTES.md#8-0-0). It is not optional although it is a significant breaking change because: - A workflow or action with random content may lead to unexpected behavior that is very difficult to figure out and debug - When something works by chance but is not documented or supported, users will start to depend on it and it will have to be supported or migrated in a backward compatible way. This is can be very difficult once the runner is stable. - Errors that are only detected at runtime by an incorrect behavior can be very difficult to relate to the root cause when it is an incorrect syntactic error in the workflow. It is much less difficult to figure out a syntactic error that is displayed in the Forgejo UI or in the output of a job that fails to run because of them. ### No more GITHUB in variable names Starting with v7.0.0, the runner allows to write new workflows [without ever using `github` in the name of contexts or variables](https://code.forgejo.org/forgejo/act/pulls/171). It is mostly cosmetic, reason why it took years to land. The previous names that include `github` are preserved for backward compatibility and to facilitate the re-use of actions that were originally developed exclusively for GitHub. Shortly after this release was deployed in the Forgejo infrastructure, a [series of pull requests](https://code.forgejo.org/forgejo/runner/issues/677) was merged to take advantage of those nicer looking names. ### Bugs and features reorganization All issues and pull requests for ACT and the runner were scrubbed and triaged. Bug reports now are grouped in [the Forgejo runner issue tracker](https://code.forgejo.org/forgejo/runner/issues) and feature requests were moved to [a dedicated space](https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues). On that occasion a [new workflow was discussed](https://codeberg.org/forgejo/discussions/issues/369) and [implemented](https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/new?template=.forgejo%2fissue_template%2ffeature-request.yaml). In a nutshell it tries to find balance between the expectations of someone filing a new feature request and the need for first hand testimonies to guide the implementation. To not re-invent the wheel, an exhaustive inventory of the commits found in the [Nektos ACT and act_runner repositories](https://code.forgejo.org/forgejo/runner/issues/678) was done and compared to the Forgejo runner implementation. It had to go back one year but there has not been a lot of activity: it did not take too long and the commits that were discovered will save valuable development time. ### Design and User Research A [design](https://codeberg.org/forgejo/design/issues/47) and user research effort ([1](https://codeberg.org/forgejo/user-research/issues/64) and [2](https://codeberg.org/forgejo/user-research/issues/63)) began to figure out how Forgejo Actions CI results should be followed. It will help figure out the next steps now that Forgejo v12.0 has the building blocks allowing it to [send mail notifications when a workflow fails](https://forgejo.org/2025-07-release-v12-0/#forgejo-actions-email-notifications-on-failure).

Beowulf commented 2025年07月27日 21:45:26 +02:00

forgejo/discussions#372

https://codeberg.org/forgejo/discussions/issues/372

earl-warren commented 2025年07月28日 10:38:29 +02:00

The following snippets were updated and are ready to be used for the monthly update (@viceice @msrd0 @christopher-besch ).

• Releases
• Infrastructure
• Forgejo Actions

The following snippets were updated and are ready to be used for the monthly update (@viceice @msrd0 @christopher-besch ). - [Releases](https://codeberg.org/forgejo/website/issues/609#issuecomment-5921563) - [Infrastructure](https://codeberg.org/forgejo/website/issues/609#issuecomment-5921740) - [Forgejo Actions](https://codeberg.org/forgejo/website/issues/609#issuecomment-5922190)

viceice commented 2025年07月28日 13:04:18 +02:00

Helm chart

Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions.
v13 is a major breaking change because we replaced Redis with Valkey. Also the PostgreSQL HA subchart had a major bump and needs manual migration if used.

https://code.forgejo.org/forgejo-helm/forgejo-helm/releases

## Helm chart Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions. v13 is a major breaking change because we replaced Redis with Valkey. Also the PostgreSQL HA subchart had a major bump and needs manual migration if used. https://code.forgejo.org/forgejo-helm/forgejo-helm/releases

earl-warren commented 2025年07月28日 16:54:42 +02:00

The short draft blog post about the conclusion of a one year long effort to improve the Forgejo runner security could be added (v8.0.0 contains the last of it, masking multiline secrets).

#566

The short draft blog post about the conclusion of a one year long effort to improve the Forgejo runner security could be added (v8.0.0 contains the last of it, masking multiline secrets). https://codeberg.org/forgejo/website/pulls/566

mahlzahn commented 2025年07月28日 16:59:48 +02:00

@viceice wrote in #609 (comment):

Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions.

v7 with Forgejo v7.0, v12 with v11.0 and v13 with v12.0? Maybe the following instead?

Some bugfix releases where made for v7 (Forgejo v7.0) and v12 (Forgejo v11.0) and v13 (Forgejo v12.0).

But, actually Forgejo v7.0 is EOL and might not get future security bug fixes.

@viceice wrote in https://codeberg.org/forgejo/website/issues/609#issuecomment-6020194: > Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions. v7 with Forgejo v7.0, v12 with v11.0 and v13 with v12.0? Maybe the following instead? > Some bugfix releases where made for v7 (Forgejo v7.0) and v12 (Forgejo v11.0) and v13 (Forgejo v12.0). But, actually Forgejo v7.0 is EOL and might not get future security bug fixes.

Beowulf commented 2025年07月28日 18:27:38 +02:00

!626 is created, please move to the PR for further things 😃.

!626 is created, please move to the PR for further things 😃.

viceice commented 2025年07月28日 19:19:44 +02:00

@mahlzahn wrote in #609 (comment):

@viceice wrote in #609 (comment):

Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions.

v7 with Forgejo v7.0, v12 with v11.0 and v13 with v12.0? Maybe the following instead?

Some bugfix releases where made for v7 (Forgejo v7.0) and v12 (Forgejo v11.0) and v13 (Forgejo v12.0).

But, actually Forgejo v7.0 is EOL and might not get future security bug fixes.

sure, chart v7 and v11 are now eol.

@mahlzahn wrote in https://codeberg.org/forgejo/website/issues/609#issuecomment-6022999: > @viceice wrote in #609 (comment): > > > Some bugfix releases where made for v7 (v7.1.13 and v7.1.14) and v12 (v12.5.3 and v12.5.4). v13 (v13.0.0 and v13.0.1) was released with corresponding Forgejo versions. > > v7 with Forgejo v7.0, v12 with v11.0 and v13 with v12.0? Maybe the following instead? > > > Some bugfix releases where made for v7 (Forgejo v7.0) and v12 (Forgejo v11.0) and v13 (Forgejo v12.0). > > But, actually Forgejo v7.0 is EOL and might not get future security bug fixes. sure, chart v7 and v11 are now eol.