forgejo/discussions
49
43

Promoting Docker actions in the Forgejo Actions ecosystem #422

Open
opened 2025年11月27日 16:48:37 +01:00 by fnetX · 21 comments
Owner
Copy link

Actions for Forgejo Actions can have two types: NodeJS and Docker actions.

https://codeberg.org/umglurf/reuse-action is the first Docker action that I observed in the wild, and I investigated more on how it works on the runner side. Effectively it looks like this (from observation only, please correct me if I'm wrong):

  • The docker container used for the Action is created and cached.
  • The container is spawned and gets the volume of the existing container mounted
  • The action is executed

This has a lot of advantages over the NodeJS actions:

  • The dependencies do not need to be installed all the time. Since the container is cached, this is more lightweight than installing everything from scratch.
  • There can be different base images for different steps in the workflow, e.g. building the frontend of an application could be done using a different base image than the backend.
  • The dependency on NodeJS could be avoided: As far as I understand, there could be a checkout action that simply uses a container image that has Git and other dependencies installed (compared to the current requirement to have NodeJS available in every container image just for the checkout step)

I think it would make sense to collect more feedback on the potential of Docker actions and consider promoting them in the Forgejo Actions ecosystem, effectively working around the current limitations of the NodeJS-based Actions ecosystem. What do you think?

Actions for Forgejo Actions can have two types: NodeJS and Docker actions. https://codeberg.org/umglurf/reuse-action is the first Docker action that I observed in the wild, and I investigated more on how it works on the runner side. Effectively it looks like this (from observation only, please correct me if I'm wrong): * The docker container used for the Action is created and cached. * The container is spawned and gets the volume of the existing container mounted * The action is executed This has a lot of advantages over the NodeJS actions: * The dependencies do not need to be installed all the time. Since the container is cached, this is more lightweight than installing everything from scratch. * There can be different base images for different steps in the workflow, e.g. building the frontend of an application could be done using a different base image than the backend. * The dependency on NodeJS could be avoided: As far as I understand, there could be a checkout action that simply uses a container image that has Git and other dependencies installed (compared to the current requirement to have NodeJS available in every container image just for the checkout step) I think it would make sense to collect more feedback on the potential of Docker actions and consider promoting them in the Forgejo Actions ecosystem, effectively working around the current limitations of the NodeJS-based Actions ecosystem. What do you think?

Actions for Forgejo Actions can have two types: NodeJS and Docker actions.

and shell (AKA composite for a reason that eludes me). Forgejo uses a considerable amount of those, as you know 😁.

> Actions for Forgejo Actions can have two types: NodeJS and Docker actions. and shell (AKA composite for a reason that eludes me). Forgejo uses a considerable amount of those, as you know 😁.

For optimization purposes (not installing everything every time) an alternative approach is to use a pre-populated image rather than a Docker action, because a docker action will only populate the container it runs and the next step won't have those if it is a shell script.

For optimization purposes (not installing everything every time) an alternative approach is to use a pre-populated image rather than a Docker action, because a docker action will only populate the container it runs and the next step won't have those if it is a shell script.

there could be a checkout action that simply uses a container image that has Git and other dependencies installed

I think that would be a very welcome addition to the ecosystem. A "checkout" action that is Docker based and API compatible with the existing one and does not rely on NodeJS. I also think that it will ultimately be less work than ensuring Forgejo Actions stays compatible with whatever GitHub decides to do with the next major version of this action.

> there could be a checkout action that simply uses a container image that has Git and other dependencies installed I think that would be a very welcome addition to the ecosystem. A "checkout" action that is Docker based and API compatible with the existing one and does not rely on NodeJS. I also think that it will ultimately be less work than ensuring Forgejo Actions stays compatible with whatever GitHub decides to do with the next major version of this action.
Author
Owner
Copy link

I know that prebuilt images are an option, but they have two challenges:

  • They are more challenging to set up: You need to create a scheduled action that regularly builds them and pushes them back to a registry.
  • If such an image is not used, it is still regularly built and pushed, so you need to keep track of which prebuilt images are still in (regular) use.

The Docker Action are created ondemand and cached automatically. They are very easy to use.

Especially the NodeJS dependency is in many cases only used for actions. I'm mostly thinking about the current situation:

  • Someone builds something and would like to use a dedicated image (think python, rust, go, ...)
  • The person needs to checkout the code at the beginning and push some artifact at the end
  • For these two Actions, NodeJS becomes a dependency and either needs to be installed into the dedicated image, or a NodeJS base image is used and the other tools (think python, rust, go, ...) are installed into the NodeJS image.
  • With the Docker Actions, there checkout and artifact upload steps could be done using a NodeJS image, while the rest of the build happens in the dedicated image.

I feel like Forgejo itself is a power user of Forgejo Actions with very specific needs and it works very well. However, after observing how Forgejo Actions is used after promoting it more on Codeberg, I see that a lot of users struggle with the problem of the NodeJS dependency, especially for code checkout. And especially for small side and hobby projects, the effort to create prebuilt images just to ... get the code before you build your app, is really insane.

I know that prebuilt images are an option, but they have two challenges: * They are more challenging to set up: You need to create a scheduled action that regularly builds them and pushes them back to a registry. * If such an image is not used, it is still regularly built and pushed, so you need to keep track of which prebuilt images are still in (regular) use. The Docker Action are created ondemand and cached automatically. They are very easy to use. Especially the NodeJS dependency is in many cases only used for actions. I'm mostly thinking about the current situation: * Someone builds something and would like to use a dedicated image (think python, rust, go, ...) * The person needs to checkout the code at the beginning and push some artifact at the end * For these two Actions, NodeJS becomes a dependency and either needs to be installed into the dedicated image, or a NodeJS base image is used and the other tools (think python, rust, go, ...) are installed into the NodeJS image. * With the Docker Actions, there checkout and artifact upload steps could be done using a NodeJS image, while the rest of the build happens in the dedicated image. I feel like Forgejo itself is a power user of Forgejo Actions with very specific needs and it works very well. However, after observing how Forgejo Actions is used after promoting it more on Codeberg, I see that a lot of users struggle with the problem of the NodeJS dependency, especially for code checkout. And especially for small side and hobby projects, the effort to create prebuilt images just to ... get the code before you build your app, is really insane.
Author
Owner
Copy link

A potentially nice addition to the runner would be to allow running the GitHub ecosystem actions in a custom container. So that instead of recreating the Actions using Docker, we allow using the existing Actions and run it in a Docker container.

jobs:
 reuse:
 runs-on: codeberg-tiny
 container:
 image: golang:latest
 steps:
 - uses: actions/checkout@v4
 container: code.forgejo.org/oci/checkout:latest # the checkout action is spawned in a nodejs container with Git preinstalled (assuming that such an image is provided for convenience by Forgejo somehow)
 - run: ... # this is executed in the golang image again
 - uses: actions/upload-artifact@v4
 container: nodejs # this action is spawned in a nodejs container
A potentially nice addition to the runner would be to allow running the GitHub ecosystem actions in a custom container. So that instead of recreating the Actions using Docker, we allow using the existing Actions and run it in a Docker container. ~~~ jobs: reuse: runs-on: codeberg-tiny container: image: golang:latest steps: - uses: actions/checkout@v4 container: code.forgejo.org/oci/checkout:latest # the checkout action is spawned in a nodejs container with Git preinstalled (assuming that such an image is provided for convenience by Forgejo somehow) - run: ... # this is executed in the golang image again - uses: actions/upload-artifact@v4 container: nodejs # this action is spawned in a nodejs container ~~~

How would this work for actions that affect the "OS layer", e.g. that install and set up language runtimes in /usr? As far as I'm aware, a step container doesn't even get access to the job container's base layer, since that's not a volume mount.

How would this work for actions that affect the "OS layer", e.g. that install and set up language runtimes in `/usr`? As far as I'm aware, a step container doesn't even get access to the job container's base layer, since that's not a volume mount.

@fnetX wrote in #422 (comment):

A potentially nice addition to the runner would be to allow running the GitHub ecosystem actions in a custom container.

This is an interesting idea. Instead of adding a feature to the runner it could simply be done by embedding the current checkout action in a container, publishing it under forgejo-checkout and advertise it as an alternative that does not require NodeJS. It is a low hanging fruit. Same with the artifact actions, with the limitation that whatever it does needs to be in the workspace, which is not 100% transparent.

With the added benefit that it could be re-implemented in the future to no longer depend on NodeJS in a transparent way.

@fnetX wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-8546136: > A potentially nice addition to the runner would be to allow running the GitHub ecosystem actions in a custom container. This is an interesting idea. Instead of adding a feature to the runner it could simply be done by embedding the current `checkout` action in a container, publishing it under `forgejo-checkout` and advertise it as an alternative that does not require NodeJS. It is a low hanging fruit. Same with the artifact actions, with the limitation that whatever it does needs to be in the workspace, which is not 100% transparent. With the added benefit that it could be re-implemented in the future to no longer depend on NodeJS in a transparent way.
Author
Owner
Copy link

How would this work for actions that affect the "OS layer",

It would not work for this use case, but for a lot of others.

e.g. that install and set up language runtimes in /usr?

These actions would likely become obsolete if you are no longer forced to use nodejs as the base image. Just use the official container for the language runtime you'd like to use.

> How would this work for actions that affect the "OS layer", It would not work for this use case, but for a lot of others. > e.g. that install and set up language runtimes in /usr? These actions would likely become obsolete if you are no longer forced to use nodejs as the base image. Just use the official container for the language runtime you'd like to use.

@fnetX wrote in #422 (comment):

These actions would likely become obsolete if you are no longer forced to use nodejs as the base image. Just use the official container for the language runtime you'd like to use.

I disagree. If it was that simple, you could already do it today, by using the language-specific base image, installing nodejs for actions at runtime and then skipping the setup-X actions altogether.

But their usefulness is the stuff they do on top. Afterall, the GitHub Actions VMs these have been written for already include a lot of language runtimes pre-installed.

For example, actions-rust-lang/setup-rust-toolchain allows choosing any permutation between the various targets and toolchains Rust offers. That's not available as a pre-configured image.
Additionally, most of them, like actions/setup-node, actions/setup-go and actions-rust-lang/setup-rust-toolchain, set up caching for their respective package manager. And many of these package managers put their stuff in directories outside the workspace, that wouldn't be available to a Docker action.

@fnetX wrote in #422 (comment):

It would not work for this use case, but for a lot of others.

The problem is, as long as I need at least one JS action, I need nodejs. And if I need it anyways, why use a Docker-action-that-wraps-a-JS-action if I can just use the JS action directly for less overhead and complexity?
So you need a critical mass of supported use cases covered for this to gain serious traction. And "set up language tools" is a pretty basic use case, I'd say.

E.g. for the case with actions-rust-lang/setup-rust-toolchain, toolchain installation could reasonably be done with a Rust base image and manual rustup calls. But then I would at least need a Docker-based action to implement the caching. That's the thing that saves the bulk of CI run time.

@fnetX wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-8566119: > These actions would likely become obsolete if you are no longer forced to use nodejs as the base image. Just use the official container for the language runtime you'd like to use. I disagree. If it was that simple, you could already do it today, by using the language-specific base image, installing nodejs for actions at runtime and then skipping the `setup-X` actions altogether. But their usefulness is the stuff they do on top. Afterall, the GitHub Actions VMs these have been written for already include a lot of language runtimes pre-installed. For example, `actions-rust-lang/setup-rust-toolchain` allows choosing any permutation between the various targets and toolchains Rust offers. That's not available as a pre-configured image. Additionally, most of them, like `actions/setup-node`, `actions/setup-go` and `actions-rust-lang/setup-rust-toolchain`, set up caching for their respective package manager. And many of these package managers put their stuff in directories outside the workspace, that wouldn't be available to a Docker action. @fnetX wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-8566119: > It would not work for this use case, but for a lot of others. The problem is, as long as I need at least one JS action, I need nodejs. And if I need it anyways, why use a Docker-action-that-wraps-a-JS-action if I can just use the JS action directly for less overhead and complexity? So you need a critical mass of supported use cases covered for this to gain serious traction. And "set up language tools" is a pretty basic use case, I'd say. E.g. for the case with `actions-rust-lang/setup-rust-toolchain`, toolchain installation could reasonably be done with a Rust base image and manual `rustup` calls. But then I would at least need a Docker-based action to implement the caching. That's the thing that saves the bulk of CI run time.

I've had this thread sitting in my inbox since it was originated and have been hoping to contribute meaningfully, but have struggled... and I think the reason is that the scope of the problems vary quite a bit for different people, so there will always be reasons like @sclu1034 has raised why a single direction won't meet all needs. That variety is fine, but seeing feedback coming through the Matrix chat regularly, I agree with @fnetX's assessment that "a lot of users struggle with the problem of the NodeJS dependency", and I would add that it hits many people right in the face immediately because actions/checkout requires it.

Before commenting specifically on "Promoting Docker actions in the Forgejo Actions ecosystem", I think we need to weigh the costs and benefits to increasing the scope of the Forgejo project by developing and supporting our own actions, rather than just using (& mirroring) GitHub's.

The potential benefits we could achieve, if we did, are:

  • Simplify the environmental requirements, allowing actions to be used in more situations (used without node.js, used in host-based executors with unexpected paths)
  • Stop being limited by technical limitations that GitHub's Actions have (such as a lack of support for SHA256 repos, or the lack of support for non-github.com servers in upload-artifact and download-artifact)
  • Ability to add new features (quite a bit of discussion recently about weak dependency management in GitHub Actions, for example)

But the potential cost is a spreading of contributor effort over "more things". This cost could be managed by keeping "more things" to a very constrained set -- for example, an alternative actions/checkout which is used by 95% of actions might meet the benefit vs. cost bar, but an alternative actions-rust-lang/setup-rust-toolchain is too niche for it to be part of the Forgejo's project area of responsibility.

To circle back to the question of Docker actions... I do think it's a pretty good idea, and if we were going to self develop/host/maintain actions I think it sounds like the right way to go. But I would describe the technical capability as it exists today as "rudimentary":

  • I've just documented a long-standing bug that is relevant to be aware of in this area; I think this is a big annoyance, almost blocker, for wide-spread usage.
  • I believe that these actions are effectively unusable in a host-based runner today. They act as if they're going to be sharing a volume mount with the rest of the job, but the rest of the job isn't a container, so they get no access to any working directories.
  • I'm skeptical of the image: Dockerfile approach with a build cache because it can result in different hosts having different images -- eg. apt-get update && apt-get install will vary between each image depending on the time it was built. However, image: docker://registry.example.com/some-org/some-image:somesha avoids this problem. An action template/pattern which builds a container at CI time and publishes an action would be manageable with this approach.

I think that for a lot of simple actions, a composite action with a small amount of scripting is probably quite a plausible simpler approach... for example, actions/checkout would be a super simple composite action if we didn't support all the same options, and we relied on the base container / host environment to already have git installed. Installing git is quite a bit more reasonable than node.

I've had this thread sitting in my inbox since it was originated and have been hoping to contribute meaningfully, but have struggled... and I think the reason is that the scope of the problems vary quite a bit for different people, so there will always be reasons like @sclu1034 has raised why a single direction won't meet all needs. That variety is fine, but seeing feedback coming through the Matrix chat regularly, I agree with @fnetX's assessment that "a lot of users struggle with the problem of the NodeJS dependency", and I would add that it hits many people right in the face immediately because `actions/checkout` requires it. Before commenting specifically on "Promoting Docker actions in the Forgejo Actions ecosystem", I think we need to weigh the costs and benefits to increasing the scope of the Forgejo project by developing and supporting our own actions, rather than just using (& mirroring) GitHub's. The potential benefits we could achieve, if we did, are: - Simplify the environmental requirements, allowing actions to be used in more situations (used without node.js, used in host-based executors with unexpected paths) - Stop being limited by technical limitations that GitHub's Actions have (such as a [lack of support for SHA256 repos](https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/64), or the lack of support for non-github.com servers in upload-artifact and download-artifact) - Ability to add new features (quite a bit of discussion recently about weak dependency management in GitHub Actions, for example) But the potential cost is a spreading of contributor effort over "more things". This cost could be managed by keeping "more things" to a very constrained set -- for example, an alternative `actions/checkout` which is used by 95% of actions might meet the benefit vs. cost bar, but an alternative `actions-rust-lang/setup-rust-toolchain` is too niche for it to be part of the Forgejo's project area of responsibility. To circle back to the question of Docker actions... I do think it's a pretty good idea, and if we were going to self develop/host/maintain actions I think it sounds like the right way to go. But I would describe the technical capability as it exists today as "rudimentary": - I've just documented a [long-standing bug](https://code.forgejo.org/forgejo/runner/issues/1233) that is relevant to be aware of in this area; I think this is a big annoyance, almost blocker, for wide-spread usage. - I believe that these actions are effectively unusable in a host-based runner today. They act as if they're going to be sharing a volume mount with the rest of the job, but the rest of the job isn't a container, so they get no access to any working directories. - I'm skeptical of the `image: Dockerfile` approach with a build cache because it can result in different hosts having different images -- eg. `apt-get update && apt-get install` will vary between each image depending on the time it was built. However, `image: docker://registry.example.com/some-org/some-image:somesha` avoids this problem. An action template/pattern which builds a container at CI time and publishes an action would be manageable with this approach. I think that for a lot of simple actions, a composite action with a small amount of scripting is probably quite a plausible simpler approach... for example, `actions/checkout` would be a super simple composite action if we didn't support all the same options, and we relied on the base container / host environment to already have `git` installed. Installing `git` is quite a bit more reasonable than node.

Amongst other things, CI should be secure, reproducible, and support as many platforms as possible. Actions fail at least at the first two aspects. While using containers might improve security and reproducibility, they would be a step backwards when it comes to portability. Running containers on BSD and Windows is problematic at best. There also seem to be licensing issues.

For those reasons alone, I don't think we should go there.

Instead, we should try to figure out how we could make actions entirely optional. How can we check code out, download and collect artefacts, or maintain a cache without them? It would have the added benefit that Forgejo Actions keeps working even if something happened to GitHub Actions.

For everything else, nobody stops users from installing arbitrary binaries or packages with the package manager of their choice. That might not sound user-friendly at first, but is a much better solution overall. It leverages existing technologies, is not bound to a particular CI service, and much easier to test locally.

Amongst other things, CI should be secure, reproducible, and support as many platforms as possible. Actions [fail](https://nesbitt.io/2025/12/06/github-actions-package-manager.html) at least at the first two aspects. While using containers might improve security and reproducibility, they would be a step backwards when it comes to portability. Running containers on BSD and Windows is problematic at best. There also seem to be [licensing issues](https://github.com/actions/runner-images/issues/17#issuecomment-614726536). For those reasons alone, I don't think we should go there. Instead, we should try to figure out how we could make actions entirely optional. How can we check code out, download and collect artefacts, or maintain a cache without them? It would have the added benefit that Forgejo Actions keeps working even if something happened to GitHub Actions. For everything else, nobody stops users from installing arbitrary binaries or packages with the package manager of their choice. That might not sound user-friendly at first, but is a much better solution overall. It leverages existing technologies, is not bound to a particular CI service, and much easier to test locally.

If somebody stumbles on the same issue that uses: actions/checkout@v4 fails due to missing NodeJS in the Docker image,
I just created a simple reusable action for this: https://codeberg.org/Tienisto/checkout 

- name:Checkoutuses:https://codeberg.org/Tienisto/checkout@v0.1.0

It does need git to be installed though and should only work for common setups.

I needed it for https://codeberg.org/Tienisto/slang/src/branch/main/.forgejo/workflows/ci.yml

If somebody stumbles on the same issue that `uses: actions/checkout@v4` fails due to missing NodeJS in the Docker image, I just created a simple reusable action for this: https://codeberg.org/Tienisto/checkout ```yaml - name: Checkout uses: https://codeberg.org/Tienisto/checkout@v0.1.0 ``` It does need `git` to be installed though and should only work for common setups. I needed it for https://codeberg.org/Tienisto/slang/src/branch/main/.forgejo/workflows/ci.yml

Having some sort of builtin actions sounds great on paper. But where does one stop? It's difficult to tell. This sort of approach very quickly leads to a feature explosion and one needs to make a decision to stop at a seemingly arbitrary point.

If the "plugin action" (for lack of a better name) runs in the runner process itself this also significantly increases the attack surface on the runner. So these should IMO still be sandboxed the same way as any other script / action that is being run.

But if the runner basically just copies a prebuilt binary in and executes it, my first thought is "how can I make my own out of tree"?

This train of thought basically leads to "actions" that are neither shell scripts (which are rather difficult to maintain in general) nor node, but still easily executable without many or any dependencies in the runner image.

The does not leave much tbh. Python (or any other scripting language) seems like a side-step at best. Pre-built portable binaries created by the actions authors could work, but then you have to rely on them building a binary for your architecture. But those would probably be the most portable, and at least the first party ones could be built for the same architectures as the runner itself.

Having some sort of builtin actions sounds great on paper. But where does one stop? It's difficult to tell. This sort of approach very quickly leads to a feature explosion and one needs to make a decision to stop at a seemingly arbitrary point. If the "plugin action" (for lack of a better name) runs in the runner process itself this also significantly increases the attack surface on the runner. So these should IMO still be sandboxed the same way as any other script / action that is being run. But if the runner basically just copies a prebuilt binary in and executes it, my first thought is "how can I make my own out of tree"? This train of thought basically leads to "actions" that are neither shell scripts (which are rather difficult to maintain in general) nor node, but still easily executable without many or any dependencies in the runner image. The does not leave much tbh. Python (or any other scripting language) seems like a side-step at best. Pre-built portable binaries created by the actions authors could work, but then you have to rely on them building a binary for your architecture. But those would probably be the most portable, and at least the first party ones could be built for the same architectures as the runner itself.

I don't think that it's feasible or even desirable to include actions in Forgejo Runner. So far, I see three options (see below). We could do one, mix and match, or do nothing. There might be more options. In any case, support for actions would stay.

I cannot stress enough that those are ideas. I haven't tried any of those. There might be serious flaws in there.

Provide binaries

Today, Forgejo Runner downloads actions from the internet by cloning Git repositories containing actions. It makes those actions available to the respective execution environment (host, container, ...). It could do more or less the same for arbitrary binaries. That would effectively be an action replacement.

There would be a new workflow syntax, something like:

- uses-binary:https://example.com/path/to/my-program-${{os}}-${{ arch }}.xzsha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855compression:xz# or none, or zstdname:downloaded-program- run:| downloaded-program --with --args do-something

Forgejo Runner would maintain a local cache. We would still have to investigate whether that's sufficiently secure and reproducible.

Advantages: no Node.js dependency, reproducible (because pinned binaries), no dependency on the actions format, no lock-in (anybody can provide binaries), can be tested without a runner.

Disadvantages: binaries must be a single file, some programming languages cannot be used, binaries must be portable, does not solve the actions/checkout problem

Move essential functionality to Forgejo Runner

It might be possible or even sensible to delegate some essential functionality to Forgejo Runner. When I talk about essential functionality, then I mean actions that are required to get anything going:

  • actions/checkout
  • actions/upload-artifact
  • actions/download-artifact
  • actions/cache (?)

Anything else is non-essential.

It would require extending the workflow syntax with some magic steps. Example:

- collect-artifactsname:artifacts.zipcompress:zippaths:- /path/to/artifacts/**

Forgejo Runner would recognize those steps and, for example, collect and upload the artefacts itself.

The biggest challenge is actions/checkout. Forgejo Runner already performs Git operations and knows how to provide repositories to the execution environment. But interoperability can be tricky. I also don't know how secure that would be.

Big question: how to scan files in a container that might lack all necessary tools?

Advantages: basic workflows no longer require actions, no Node.js dependency for essential operations

Disadvantages: maintenance burden; potentially less functionality than existing actions.

Provide replacements for essential actions

When I talk about essential functionality, then I mean actions that are required to get anything going:

  • actions/checkout
  • actions/upload-artifact
  • actions/download-artifact
  • actions/cache (?)

Advantages: Forgejo Runner doesn't have to be changed, no new workflow syntax

Disadvantages: maintenance burden, potentially less functionality than existing actions, fundamental problems of actions (testability, reusability, reproducibility, security) remain

I don't think that it's feasible or even desirable to include actions in Forgejo Runner. So far, I see three options (see below). We could do one, mix and match, or do nothing. There might be more options. In any case, support for actions would stay. I cannot stress enough that those are ideas. I haven't tried any of those. There might be serious flaws in there. ## Provide binaries Today, Forgejo Runner downloads actions from the internet by cloning Git repositories containing actions. It makes those actions available to the respective execution environment (host, container, ...). It could do more or less the same for arbitrary binaries. That would effectively be an action replacement. There would be a new workflow syntax, something like: ```yaml - uses-binary: https://example.com/path/to/my-program-${{os}}-${{ arch }}.xz sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 compression: xz # or none, or zstd name: downloaded-program - run: | downloaded-program --with --args do-something ``` Forgejo Runner would maintain a local cache. We would still have to investigate whether that's sufficiently secure and reproducible. Advantages: no Node.js dependency, reproducible (because pinned binaries), no dependency on the actions format, no lock-in (anybody can provide binaries), can be tested without a runner. Disadvantages: binaries must be a single file, some programming languages cannot be used, binaries must be portable, does not solve the `actions/checkout` problem ## Move essential functionality to Forgejo Runner It might be possible or even sensible to delegate some essential functionality to Forgejo Runner. When I talk about essential functionality, then I mean actions that are required to get anything going: * actions/checkout * actions/upload-artifact * actions/download-artifact * actions/cache (?) Anything else is non-essential. It would require extending the workflow syntax with some magic steps. Example: ```yaml - collect-artifacts name: artifacts.zip compress: zip paths: - /path/to/artifacts/** ``` Forgejo Runner would recognize those steps and, for example, collect and upload the artefacts itself. The biggest challenge is `actions/checkout`. Forgejo Runner already performs Git operations and knows how to provide repositories to the execution environment. But interoperability can be tricky. I also don't know how secure that would be. Big question: how to scan files in a container that might lack all necessary tools? Advantages: basic workflows no longer require actions, no Node.js dependency for essential operations Disadvantages: maintenance burden; potentially less functionality than existing actions. ## Provide replacements for essential actions When I talk about essential functionality, then I mean actions that are required to get anything going: * actions/checkout * actions/upload-artifact * actions/download-artifact * actions/cache (?) Advantages: Forgejo Runner doesn't have to be changed, no new workflow syntax Disadvantages: maintenance burden, potentially less functionality than existing actions, fundamental problems of actions (testability, reusability, reproducibility, security) remain

@aahlenst Aren't "Provide replacements for essential actions" and "Provide binaries" mostly complementary ideas that aren't really an either/or choice? The latter is in contrast to "move essential functionality to forgejo runner", but I see no reason why eg. the replacements for essential actions can't be eg. binaries?

@aahlenst Aren't "Provide replacements for essential actions" and "Provide binaries" mostly complementary ideas that aren't really an either/or choice? The latter is in contrast to "move essential functionality to forgejo runner", but I see no reason why eg. the replacements for essential actions can't be eg. binaries?

@fina wrote in #422 (comment):

I see no reason why eg. the replacements for essential actions can't be eg. binaries?

They can! Trying it is already on my to-do list. I only have to pick a suitable programming language for writing that binary. Needs to be something that can link libgit2 statically, is memory-safe, and easy to cross-compile. Bonus points for not needing bindings. If somebody wants to help, let me know. I'm always happy to collaborate.

@fina wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-11604085: > I see no reason why eg. the replacements for essential actions can't be eg. binaries? They can! Trying it is already on my to-do list. I only have to pick a suitable programming language for writing that binary. Needs to be something that can link [libgit2](https://libgit2.org/) statically, is memory-safe, and easy to cross-compile. Bonus points for not needing bindings. If somebody wants to help, let me know. I'm always happy to collaborate.

As a new user of Forgejo and Forgejo actions I'd like to second the assessment that the nodejs dependency is something that I bounced off of quite hard.

My apologies for the rather long response, and this is definitely more on the side of 'weigh[ing] the costs and benefits to [...] supporting our own actions' as @mfenniak puts it, with some observations about Docker actions on the side, so if it needs to be relocated or split into its own discussion post I'm happy to do that.

Some context for my own use-case, and some additional thoughts that come out of that:

I'm currently developing a fork of a GCC-based toolchain targeting specific hardware. I currently build and consume my own MSYS2 packages for this toolchain on Windows.
Cross-compiling this toolchain from Linux is not feasible, as it needs to be built against MSYS2-compatible packages all the way down and I don't know that rebuilding the world using a mingw-cross-compiler is sufficient to maintain that level of compatibility. Official MSYS/MINGW packages are normally built using Github Windows runners, but I am trying to move away from GH rather than taking an additional dependency.
As a result, I have a bare-metal Windows box I intend to use as a build agent, controlled by a self-hosted Forgejo instance on a Linux VPS.
Because this Windows box is physical hardware in my control, running a non-server SKU, only running my own CI jobs, I don't need the additional overhead of a VM being run inside it, nor do I want to step into the license quagmire of Windows native containers. This implies my runner will be operating in host mode. Installing nodejs just to check out the repository and push artifacts to forgejo simply takes up more room on my device while also increasing the potential attack surface.

Bearing all that in mind:

  • While I understand that Windows support is not a priority for Forgejo maintainers, a dependency on Docker for minimum functionality is just as bad if not worse than a nodejs install. One of Forgejo's selling points is its lightweight nature, and having to spin up a VM (as it would be on Windows, if not using license-encumbered native containers) just to clone a repository seems to be inconsistent with that.

  • I'd be more inclined toward the 'just go roll your own actions without the runtime dependency' option (at least for everything aside from checkout/clone, which is so fundamental I'd argue that a basic version of it should be built-in regardless) if the endpoints for that bare-minimum functionality were documented. For example, information is provided on how to upload generic packages to the package registry, but the endpoints for pushing artifacts from a workflow run are, as best I can tell, undocumented.
    As I mention when discussing the PoC linked below, regardless of using Forgejo actions or another CI solution, implementing your own artifact upload and download actions, currently relies on assuming complete equivalency of Github and Forgejo Actions APIs, and then reverse-engineering implementations aimed at Github to understand what the endpoints and parameters are. Documenting the APIs used by Forgejo actions, would allow developers with needs that aren't met by whatever implementation you choose, to either integrate with another CI solution or to roll their own actions that suit their deployment environment.

  • Regarding the advantages/disadvantages of supporting forgejo-native actions that don't simply emulate Github ones, I know there's been some discussion of providing simpler APIs for interacting with artifacts, eg forgejo/forgejo#7926 . These APIs would be by necessity native to Forgejo without being bound to compatibility with Github's implementation. While there are valid concerns about spreading the development capacity of the team too thinly, I think that maintaining forgejo-native actions that are tied to forgejo-specific APIs will result in a more predictable maintenance burden in the long term - if Github drops support for old APIs or changes the behaviour of the actions that Forgejo-actions are depending on, does that not force Forgejo to make unplanned changes to remain compatible at the API level, on Github's timetable? I dont believe Github makes any substantial guarantees as to the behaviour of the internal APIs that the actions are talking to. If such a change in GH's internal API results in Forgejo actions still consuming a version of upload-artifact that Github no longer supports, for example, because Forgejo's internal API still mirrors that older interface, Forgejo is effectively no longer fully Github compatible anyway.

  • In theory, the current advantage of a single-binary runner is just that - it's a single binary that can be dropped anywhere. I don't know that there's a great deal of advantage to providing those core 3 actions that many people will want as separate binaries, all you're doing is making deployment more difficult by requiring the distribution of multiple files instead of one, and by potentially selecting another language for the implementation of those essential actions, you're further increasing development complexity by introducing another toolchain.

  • Moving beyond the bare minimum functionality, rather than implementing other actions within the Forgejo project itself, I do think there is merit to considering the notion of 'plugins' which are essentially binaries that the runner communicates with. This is the approach that Crow CI uses for example - they provide builtin clone functionality as part of the single runner binary, along with a plugin interface. Teams can then extend builtin functionality in whatever language or toolchain makes the most sense for them. Crow's implementation assumes the plugins are resident in a container, though, which I would regard as an unfriendly choice given the disadvantages provided up-thread for not providing builtins as containers.

  • As a result of the above, my feedback as a new user, would be to suggest integrating the bare minimum (checkout, upload/download artifact) directly into the runner, along with ensuring the APIs for Actions functionality (eg artifact upload/download, cache, etc) are well documented so that people can build their own actions with the choice of dependencies or runtimes that is best for them. As a future enhancement, it's worth considering a mechanism for additional binary actions that can be preinstalled alongside the runner by the instance administrator. Keeping the scope of builtin actions to an absolute minimum is a valid concern, but a hard-and-fast policy about what constitutes a core action, and therefore should be a builtin, is more likely to 'stick' if it's as easy as possible to implement extended functionality elsewhere.

I have a proof-of-concept implementation of builtin clone and artifact upload at https://code.forgejo.org/stephenwhittle/runner/src/branch/feat-add_builtin_actions.

They don't support the full set of Github-compatible options, but I've done testing on a private Forgejo instance with the following snippet, and it works correctly:

steps:- name:checkout codeuses:builtin/clone- name:build msys2 packagerun:| git config --global core.autocrlf input
 ./rebuild_pkg.sh- name:upload packageuses:builtin/upload-artifactwith:path:"*.pkg.tar.zst"archive:falsename:"mipsel-none-elf-binutils-${{ github.sha }}.pkg.tar.zst"

As I say, this is mostly a hacky proof-of-concept, with the caveats that not every function of those existing APIs has been implemented, and I am not a Go developer, in fact this is my first experience with the language, so I am likely not using idiomatic Go, and there are probably a lot of project-specific conventions I'm missing.
For the checkout action I was able to reuse the existing clone methods the runner uses for fetching actions, but for uploading artifacts, it was much more difficult - I had to rely on notes buried in discussions about the Github Actions implementation to understand the API in question - so that's quite rough-and-ready, doesn't handle HTTP errors well, etc.
That said, I figured if I was going to advocate for builtins, I should demonstrate how such a builtin implementation could work, as something concrete to keep the conversation going.

As a new user of Forgejo and Forgejo actions I'd like to second the assessment that the nodejs dependency is something that I bounced off of quite hard. My apologies for the rather long response, and this is definitely more on the side of 'weigh[ing] the costs and benefits to [...] supporting our own actions' as @mfenniak puts it, with some observations about Docker actions on the side, so if it needs to be relocated or split into its own discussion post I'm happy to do that. Some context for my own use-case, and some additional thoughts that come out of that: I'm currently developing a fork of a GCC-based toolchain targeting specific hardware. I currently build and consume my own MSYS2 packages for this toolchain on Windows. Cross-compiling this toolchain from Linux is not feasible, as it needs to be built against MSYS2-compatible packages all the way down and I don't know that rebuilding the world using a mingw-cross-compiler is sufficient to maintain that level of compatibility. Official MSYS/MINGW packages are normally built using Github Windows runners, but I am trying to move away from GH rather than taking an additional dependency. As a result, I have a bare-metal Windows box I intend to use as a build agent, controlled by a self-hosted Forgejo instance on a Linux VPS. Because this Windows box is physical hardware in my control, running a non-server SKU, only running my own CI jobs, I don't need the additional overhead of a VM being run inside it, nor do I want to step into the license quagmire of Windows native containers. This implies my runner will be operating in host mode. Installing nodejs just to check out the repository and push artifacts to forgejo simply takes up more room on my device while also increasing the potential attack surface. Bearing all that in mind: * While I understand that Windows support is not a priority for Forgejo maintainers, a dependency on Docker for minimum functionality is just as bad if not worse than a nodejs install. One of Forgejo's selling points is its lightweight nature, and having to spin up a VM (as it would be on Windows, if not using license-encumbered native containers) just to clone a repository seems to be inconsistent with that. * I'd be more inclined toward the 'just go roll your own actions without the runtime dependency' option (at least for everything aside from checkout/clone, which is so fundamental I'd argue that a basic version of it should be built-in regardless) if the endpoints for that bare-minimum functionality were documented. For example, information is provided on how to upload generic packages to the package registry, but the endpoints for pushing artifacts from a workflow run are, as best I can tell, undocumented. As I mention when discussing the PoC linked below, regardless of using Forgejo actions or another CI solution, implementing your own artifact upload and download actions, currently relies on assuming complete equivalency of Github and Forgejo Actions APIs, and then reverse-engineering implementations aimed at Github to understand what the endpoints and parameters are. Documenting the APIs used by Forgejo actions, would allow developers with needs that aren't met by whatever implementation you choose, to either integrate with another CI solution or to roll their own actions that suit their deployment environment. * Regarding the advantages/disadvantages of supporting forgejo-native actions that don't simply emulate Github ones, I know there's been some discussion of providing simpler APIs for interacting with artifacts, eg https://codeberg.org/forgejo/forgejo/issues/7926 . These APIs would be by necessity native to Forgejo without being bound to compatibility with Github's implementation. While there are valid concerns about spreading the development capacity of the team too thinly, I think that maintaining forgejo-native actions that are tied to forgejo-specific APIs will result in a more predictable maintenance burden in the long term - if Github drops support for old APIs or changes the behaviour of the actions that Forgejo-actions are depending on, does that not force Forgejo to make unplanned changes to remain compatible at the API level, on Github's timetable? I dont believe Github makes any substantial guarantees as to the behaviour of the internal APIs that the actions are talking to. If such a change in GH's internal API results in Forgejo actions still consuming a version of upload-artifact that Github no longer supports, for example, because Forgejo's internal API still mirrors that older interface, Forgejo is effectively no longer fully Github compatible anyway. * In theory, the current advantage of a single-binary runner is just that - it's a single binary that can be dropped anywhere. I don't know that there's a great deal of advantage to providing those core 3 actions that many people will want as separate binaries, all you're doing is making deployment more difficult by requiring the distribution of multiple files instead of one, and by potentially selecting another language for the implementation of those essential actions, you're further increasing development complexity by introducing another toolchain. * Moving beyond the bare minimum functionality, rather than implementing other actions within the Forgejo project itself, I do think there is merit to considering the notion of 'plugins' which are essentially binaries that the runner communicates with. This is the approach that Crow CI uses for example - they provide builtin clone functionality as part of the single runner binary, along with a plugin interface. Teams can then extend builtin functionality in whatever language or toolchain makes the most sense for them. Crow's implementation assumes the plugins are resident in a container, though, which I would regard as an unfriendly choice given the disadvantages provided up-thread for not providing builtins as containers. * As a result of the above, my feedback as a new user, would be to suggest integrating the bare minimum (checkout, upload/download artifact) directly into the runner, along with ensuring the APIs for Actions functionality (eg artifact upload/download, cache, etc) are well documented so that people can build their own actions with the choice of dependencies or runtimes that is best for them. As a future enhancement, it's worth considering a mechanism for additional binary actions that can be preinstalled alongside the runner by the instance administrator. Keeping the scope of builtin actions to an absolute minimum is a valid concern, but a hard-and-fast policy about what constitutes a core action, and therefore should be a builtin, is more likely to 'stick' if it's as easy as possible to implement extended functionality elsewhere. I have a proof-of-concept implementation of builtin clone and artifact upload at https://code.forgejo.org/stephenwhittle/runner/src/branch/feat-add_builtin_actions. They don't support the full set of Github-compatible options, but I've done testing on a private Forgejo instance with the following snippet, and it works correctly: ```yml steps: - name: checkout code uses: builtin/clone - name: build msys2 package run: | git config --global core.autocrlf input ./rebuild_pkg.sh - name: upload package uses: builtin/upload-artifact with: path: "*.pkg.tar.zst" archive: false name: "mipsel-none-elf-binutils-${{ github.sha }}.pkg.tar.zst" ``` As I say, this is mostly a hacky proof-of-concept, with the caveats that not every function of those existing APIs has been implemented, and I am not a Go developer, in fact this is my first experience with the language, so I am likely not using idiomatic Go, and there are probably a lot of project-specific conventions I'm missing. For the checkout action I was able to reuse the existing clone methods the runner uses for fetching actions, but for uploading artifacts, it was much more difficult - I had to rely on notes buried in discussions about the Github Actions implementation to understand the API in question - so that's quite rough-and-ready, doesn't handle HTTP errors well, etc. That said, I figured if I was going to advocate for builtins, I should demonstrate how such a builtin implementation *could* work, as something concrete to keep the conversation going.

@stephenwhittle wrote in #422 (comment):

While I understand that Windows support is not a priority for Forgejo maintainers

I'm one of the (very few) maintainers of Forgejo Actions and Forgejo Runner. Cross-platform support is very high on my list of priorities, as is doing something about being dependent on GitHub for foundational actions which Microsoft has more or less given up on.

Cross-platform support, replacements for foundational actions, and better APIs (to a lesser extent) are all more or less predicated on solving the following problems:

  • Lack of contributors
  • Lack of a suitable testing strategy

Unfortunately, both are very hard to solve in a sustainable fashion.

@stephenwhittle wrote in #422 (comment):

As I say, this is mostly a hacky proof-of-concept, with the caveats that not every function of those existing APIs has been implemented, and I am not a Go developer, in fact this is my first experience with the language, so I am likely not using idiomatic Go, and there are probably a lot of project-specific conventions I'm missing.

This is interesting.

The biggest concern I have with the approach: The normal mode of operation for Forgejo Runner is to run outside the (confined) execution environment of a job (container, VM). That means that files have to be teleported into the execution environment and back again. That carries the risk of problems with permissions, case sensitivity, line endings, encodings, and symlinks. I wouldn't be surprised if there are more. For example: How would Forgejo Runner, running on Linux, collect artefacts on a Windows guest?

Testing Forgejo Runner isn't pleasant. Every time somebody touches code involving Git or files on the host, I have to fire up a Windows machine and verify that things still work (they break a lot on Windows). That is not sustainable. If we move more functionality into Forgejo Runner, it will only get worse. Fixing problems with built-in actions will require updating Forgejo Runner, which puts more stress on contributors.

@stephenwhittle wrote in #422 (comment):

That said, I figured if I was going to advocate for builtins, I should demonstrate how such a builtin implementation could work, as something concrete to keep the conversation going.

Thank you.

I think better APIs for uploading and downloading artefacts are within reach. If someone has a concrete proposal, please let us know. Long-term, I like to get rid of all APIs that support actions/upload-artifact and actions/download-artifact for various reasons.

@stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14936010: > While I understand that Windows support is not a priority for Forgejo maintainers I'm one of the (very few) maintainers of Forgejo Actions and Forgejo Runner. Cross-platform support is very high on my list of priorities, as is doing something about being dependent on GitHub for foundational actions which Microsoft has more or less given up on. Cross-platform support, replacements for foundational actions, and better APIs (to a lesser extent) are all more or less predicated on solving the following problems: * Lack of contributors * Lack of a suitable testing strategy Unfortunately, both are very hard to solve in a sustainable fashion. @stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14936010: > As I say, this is mostly a hacky proof-of-concept, with the caveats that not every function of those existing APIs has been implemented, and I am not a Go developer, in fact this is my first experience with the language, so I am likely not using idiomatic Go, and there are probably a lot of project-specific conventions I'm missing. This is interesting. The biggest concern I have with the approach: The normal mode of operation for Forgejo Runner is to run outside the (confined) execution environment of a job (container, VM). That means that files have to be teleported into the execution environment and back again. That carries the risk of problems with permissions, case sensitivity, line endings, encodings, and symlinks. I wouldn't be surprised if there are more. For example: How would Forgejo Runner, running on Linux, collect artefacts on a Windows guest? Testing Forgejo Runner isn't pleasant. Every time somebody touches code involving Git or files on the host, I have to fire up a Windows machine and verify that things still work (they break a lot on Windows). That is not sustainable. If we move more functionality into Forgejo Runner, it will only get worse. Fixing problems with built-in actions will require updating Forgejo Runner, which puts more stress on contributors. @stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14936010: > That said, I figured if I was going to advocate for builtins, I should demonstrate how such a builtin implementation _could_ work, as something concrete to keep the conversation going. Thank you. --- I think better APIs for uploading and downloading artefacts are within reach. If someone has a concrete proposal, please let us know. Long-term, I like to get rid of all APIs that support `actions/upload-artifact` and `actions/download-artifact` for various reasons.

The idea of builtin actions does sound tempting.

The cross-platform concerns would be real, although, it's already necessary for Runner to be able to perform some operations inside the job container. For example, running action/checkout already involves Runner fetching the repository, reading the action.yml to understand how to operate, and then copying the contents into the job container to node dist.js (or whatever). So, if using a VM-ish approach to run a Windows container with a Linux host, this is already an area that a hypothetical VM plugin will need to manage many of the problems previously mentioned.

I'm concerned that we wouldn't find a natural place to stop providing builtin plugins that would make much sense to users. The proposed minimum set (checkout, upload-artifact, download-artifact) hits "things that are definitely Forgejo Actions". But, can I checkin -- I need to edit my release notes and push them. Can I create a tag and release -- that's definitely Forgejo, using Forgejo's APIs. What do you mean I can't build a Docker container, Forgejo provides a package registry that autolinks to my repo when I push. These are all real-world use-cases and kicking users back over to the node.js ecosystem or DIY, without providing native solutions, seems kinda strange.

Considering the level of difficulty that the typical self-hosting user has with Forgejo Runner installation and configuration, plugins for more actions is a pretty complicated ask I think. For someone willing to run a k8s cluster and wants their jobs run there, sure, they're in the range of "install plugins in Forgejo Runner". But many users aren't here.

It's great to have a proof-of-concept of the idea. I'm certainly not opposed, but I think there are plausible other ideas that might work better (or worse). I think in the long-term we need to compare it to some other options, and having something built as a starting point will help with that. But @aahlenst's concerns about contributors and testing are definitely the major blockers to making that effort go.

The idea of builtin actions does sound tempting. The cross-platform concerns would be real, although, it's already necessary for Runner to be able to perform some operations inside the job container. For example, running `action/checkout` already involves Runner fetching the repository, reading the `action.yml` to understand how to operate, and then copying the contents into the job container to `node dist.js` (or whatever). So, if using a VM-ish approach to run a Windows container with a Linux host, this is already an area that a hypothetical VM plugin will need to manage *many* of the problems previously mentioned. I'm concerned that we wouldn't find a natural place to stop providing builtin plugins that would make much sense to users. The proposed minimum set (checkout, upload-artifact, download-artifact) hits "things that are definitely Forgejo Actions". But, can I checkin -- I need to edit my release notes and push them. Can I create a tag and release -- that's definitely Forgejo, using Forgejo's APIs. What do you mean I can't build a Docker container, Forgejo provides a package registry that autolinks to my repo when I push. These are all real-world use-cases and kicking users back over to the node.js ecosystem or DIY, without providing native solutions, seems kinda strange. Considering the level of difficulty that the typical self-hosting user has with Forgejo Runner installation and configuration, plugins for more actions is a pretty complicated ask I think. For someone willing to run a k8s cluster and wants their jobs run there, sure, they're in the range of "install plugins in Forgejo Runner". But many users aren't here. It's great to have a proof-of-concept of the idea. I'm certainly not opposed, but I think there are plausible other ideas that *might* work better (or worse). I think in the long-term we need to compare it to some other options, and having something built as a starting point will help with that. But @aahlenst's concerns about contributors and testing are definitely the major blockers to making that effort go.

@aahlenst: I'm one of the (very few) maintainers of Forgejo Actions and Forgejo Runner. Cross-platform support is very high on my list of priorities, as is doing something about being dependent on GitHub for foundational actions which Microsoft has more or less given up on.

My apologies - My understanding was based on discussions such as #223 and the runner readme, which mentions support only for Linux-derived operating systems, with Windows binaries available through a seemingly-unaffiliated Github project - it looked like Windows runner support was deprioritized. I'm grateful to hear that this was a faulty impression of mine.

@aahlenst: For example: How would Forgejo Runner, running on Linux, collect artefacts on a Windows guest?

My understanding is that Docker on Linux and LXC are container-only runtimes, so cannot run a Windows VM, and of course Windows containers require a Windows host.
Docker for Windows has support for both Windows containers and Linux/Windows VMs via hyper-v, but I think this implies the only way to have Windows guests is to use a Windows host and a runner in host mode, or docker mode with VM/native Windows container, so I dont think this particular combination is possible unless VM support was added for Linux hosts?
All of which is speaking more to your example, than to the actual point, though - I do acknowledge that transport of data back out of containers using the built-in action wasn't something I'd given a lot of thought to. It isn't insurmountable, but it's true that it probably requires more consideration of, and consistency with, how Docker actions (which suffer similar issues) interact with runners, especially those in 'host' mode. This includes a discussion of what the project considers an acceptable level of isolation, let alone the permissions/case sensitivity/other cross-platform issues that both of you have pointed out.

As a result of that uncertainty given cross-platform issues, separate binaries for 'core' actions is possibly a better solution if it avoids the need to extract data from containers or VMs, but if that was the option taken I think that it might be worth revisiting the assumption of git being required inside the container, though.
On one hand, making that a requirement allows fetching core actions to be performed by having the runner invoke the container-local git to clone them via JobContainer.Exec, then Exec the resulting file, and this way all downloads, etc, are being isolated inside the container.
However, the current behaviour of the runner for actions, as I understand it, is to clone the action outside of the container and then mount/push a worktree of that action's git repository to the container, and then Exec it. Would it be feasible to do the same thing with core actions, i.e download them via the runner and then make that available to containers/VMs etc via a shared folder? That keeps communication one-way, and would eliminate the 'git-in-container' as a hard requirement unless a user wants to perform some other action requiring it.

@aahlenst: Fixing problems with built-in actions will require updating Forgejo Runner, which puts more stress on contributors.
@mfenniak: [...] concerns about contributors and testing are definitely the major blockers to making that effort go.

These are both fair points and I certainly don't wish to be cavalier about the impact on contributors.
I think there's still some potential value in considering a convenience version of clone/checkout which doesn't require an external binary, perhaps one that only supports cloning the workflow's target repository to the root of the workspace and doesn't provide other parameters. As I did in my PoC, that could be implemented as a wrapper around the existing clone method that the runner already has, i.e theres in theory only a minimum amount of additional code to maintain. If this is something that you'd consider, I'd be willing to refactor my PoC with just that single convenience implementation so that it could be tested with.
This doesn't even have to be a step but instead could be controlled by a parameter on the workflow itself, performed during job setup. At that point it's not really an action any more, rather a different feature, and framing it as a setup step might make it easier to explain why there's not a native implementation for any other 'core action' that talks to the Forgejo API.
I do acknowledge that it still involves some additional maintenance burden on the Runner codebase, and it doesn't eliminate the need for more complex checkout action as part of the 'core actions', so it isn't a zero-cost feature, but just a thought.

@aahlenst: I think better APIs for uploading and downloading artefacts are within reach. If someone has a concrete proposal, please let us know. Long-term, I like to get rid of all APIs that support actions/upload-artifact and actions/download-artifact for various reasons.

Not being a backend developer, I went looking for prior art for this. I didn't have a lot of luck, because a lot of other projects with integrated storage, don't document their APIs and want you to use their opaque upload tool or a magic action to perform the upload, and only provide a 'download' endpoint. Otherwise they don't have native storage and want you to submit to your own S3 compatible storage and make an API call to associate the resulting resource with your build as an artifact, or you specify artifacts by name in the job configuration and the CI runtime itself handles everything for you.

I think there are two sets of functionality that the artefact APIs could potentially target: one is relatively simple and equivalent to the generic package API, but ties the upload to a workflow run, and the other is a full-featured replacement for the endpoints that are consumed by actions/upload-artifact and actions/download-artifact which implements a similar feature set but with our own interface that explicitly considers GHA compatibility a non-goal.

For the former, forgejo/forgejo#7926 mentions a desire for something analogous to the existing generic package API, and we now have the other artifact endpoints from forgejo/forgejo#12140 :

Generic package is
PUT /packages/{owner}/generic/{package_name}/{package_version}/{file_name}

and the GET for a listing of artifacts is
GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts

This suggests that even a simple
PUT /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts/{file_name}
would at least have the advantage of being consistent with other new APIs that are provided.

but it really depends on what set of functionality the project actually wishes to offer, rather than things you've been obligated to offer to date because of the desire to be GHA compatible.
Things like multipart upload, or an additional endpoint that associates an artifact filename/path to external S3-compatible storage, by their very nature imply a more complex set of endpoints which is unlikely to be similar to the existing package APIs.
As I say, not a BE dev here so I don't really know that this constitutes a concrete proposal, and if my suggestion has enough merit we probably want to take it to a dedicated discussion thread of its own, but it has the advantage of simplicity and consistency at least.

@mfenniak: kicking users back over to the node.js ecosystem or DIY, without providing native solutions, seems kinda strange
@mfenniak: plugins for more actions is a pretty complicated ask

Given improvements to artifacts endpoints might be a little while coming while there is more discussion about what those new endpoints could be, it would be good to document the existing upload/download endpoints (and any other currently-undocumented endpoints I haven't encountered yet), with parameters and a cURL command like we do for the package API. This would at least facilitate the 'roll your own' option discussed up-thread. It isn't a true native solution but is probably the best thing that we have for now, and it's substantially less complicated than a plugin. I'd do the artifact ones myself, but I'm not yet familiar enough with the relevant endpoints for reviewing any submission I made to be a good use of maintainer time, I don't think.

> @aahlenst: I'm one of the (very few) maintainers of Forgejo Actions and Forgejo Runner. Cross-platform support is very high on my list of priorities, as is doing something about being dependent on GitHub for foundational actions which Microsoft has more or less given up on. My apologies - My understanding was based on discussions such as https://codeberg.org/forgejo/discussions/issues/223 and the runner readme, which mentions support only for Linux-derived operating systems, with Windows binaries available through a seemingly-unaffiliated Github project - it looked like Windows runner support was deprioritized. I'm grateful to hear that this was a faulty impression of mine. > @aahlenst: For example: How would Forgejo Runner, running on Linux, collect artefacts on a Windows guest? My understanding is that Docker on Linux and LXC are container-only runtimes, so cannot run a Windows VM, and of course Windows containers require a Windows host. Docker for Windows has support for both Windows containers and Linux/Windows VMs via hyper-v, but I think this implies the only way to have Windows guests is to use a Windows host and a runner in host mode, or docker mode with VM/native Windows container, so I dont think this particular combination is possible unless VM support was added for Linux hosts? All of which is speaking more to your example, than to the actual point, though - I do acknowledge that transport of data back out of containers using the built-in action wasn't something I'd given a lot of thought to. It isn't insurmountable, but it's true that it probably requires more consideration of, and consistency with, how Docker actions (which suffer similar issues) interact with runners, especially those in 'host' mode. This includes a discussion of what the project considers an acceptable level of isolation, let alone the permissions/case sensitivity/other cross-platform issues that both of you have pointed out. As a result of that uncertainty given cross-platform issues, separate binaries for 'core' actions is possibly a better solution if it avoids the need to extract data from containers or VMs, but if that was the option taken I think that it might be worth revisiting the assumption of `git` being required inside the container, though. On one hand, making that a requirement allows fetching core actions to be performed by having the runner invoke the container-local `git` to clone them via `JobContainer.Exec`, then Exec the resulting file, and this way all downloads, etc, are being isolated inside the container. However, the current behaviour of the runner for actions, as I understand it, is to clone the action outside of the container and then mount/push a worktree of that action's git repository to the container, and then Exec it. Would it be feasible to do the same thing with core actions, i.e download them via the runner and then make that available to containers/VMs etc via a shared folder? That keeps communication one-way, and would eliminate the '`git`-in-container' as a hard requirement unless a user wants to perform some other action requiring it. > @aahlenst: Fixing problems with built-in actions will require updating Forgejo Runner, which puts more stress on contributors. > @mfenniak: [...] concerns about contributors and testing are definitely the major blockers to making that effort go. These are both fair points and I certainly don't wish to be cavalier about the impact on contributors. I think there's still some potential value in considering a convenience version of `clone`/`checkout` which doesn't require an external binary, perhaps one that only supports cloning the workflow's target repository to the root of the workspace and doesn't provide other parameters. As I did in my PoC, that could be implemented as a wrapper around the existing clone method that the runner already has, i.e theres in theory only a minimum amount of additional code to maintain. If this is something that you'd consider, I'd be willing to refactor my PoC with just that single convenience implementation so that it could be tested with. This doesn't even have to be a step but instead could be controlled by a parameter on the workflow itself, performed during job setup. At that point it's not really an action any more, rather a different feature, and framing it as a setup step might make it easier to explain why there's not a native implementation for any other 'core action' that talks to the Forgejo API. I do acknowledge that it still involves some additional maintenance burden on the Runner codebase, and it doesn't eliminate the need for more complex checkout action as part of the 'core actions', so it isn't a zero-cost feature, but just a thought. > @aahlenst: I think better APIs for uploading and downloading artefacts are within reach. If someone has a concrete proposal, please let us know. Long-term, I like to get rid of all APIs that support `actions/upload-artifact` and `actions/download-artifact` for various reasons. Not being a backend developer, I went looking for prior art for this. I didn't have a lot of luck, because a lot of other projects with integrated storage, don't document their APIs and want you to use their opaque upload tool or a magic action to perform the upload, and only provide a 'download' endpoint. Otherwise they don't have native storage and want you to submit to your own S3 compatible storage and make an API call to associate the resulting resource with your build as an artifact, or you specify artifacts by name in the job configuration and the CI runtime itself handles everything for you. I think there are two sets of functionality that the artefact APIs could potentially target: one is relatively simple and equivalent to the generic package API, but ties the upload to a workflow run, and the other is a full-featured replacement for the endpoints that are consumed by `actions/upload-artifact` and `actions/download-artifact` which implements a similar feature set but with our own interface that explicitly considers GHA compatibility a non-goal. For the former, https://codeberg.org/forgejo/forgejo/issues/7926 mentions a desire for something analogous to the existing generic package API, and we now have the other artifact endpoints from https://codeberg.org/forgejo/forgejo/pulls/12140 : Generic package is `PUT /packages/{owner}/generic/{package_name}/{package_version}/{file_name}` and the GET for a listing of artifacts is `GET /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts` This suggests that even a simple `PUT /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts/{file_name}` would at least have the advantage of being consistent with other new APIs that are provided. but it really depends on what set of functionality the project actually wishes to offer, rather than things you've been obligated to offer to date because of the desire to be GHA compatible. Things like multipart upload, or an additional endpoint that associates an artifact filename/path to external S3-compatible storage, by their very nature imply a more complex set of endpoints which is unlikely to be similar to the existing package APIs. As I say, not a BE dev here so I don't really know that this constitutes a concrete proposal, and if my suggestion has enough merit we probably want to take it to a dedicated discussion thread of its own, but it has the advantage of simplicity and consistency at least. > @mfenniak: kicking users back over to the node.js ecosystem or DIY, without providing native solutions, seems kinda strange > @mfenniak: plugins for more actions is a pretty complicated ask Given improvements to artifacts endpoints might be a little while coming while there is more discussion about what those new endpoints could be, it would be good to document the existing upload/download endpoints (and any other currently-undocumented endpoints I haven't encountered yet), with parameters and a cURL command like we do for the package API. This would at least facilitate the 'roll your own' option discussed up-thread. It isn't a true native solution but is probably the best thing that we have for now, and it's substantially less complicated than a plugin. I'd do the artifact ones myself, but I'm not yet familiar enough with the relevant endpoints for reviewing any submission I made to be a good use of maintainer time, I don't think.

@stephenwhittle wrote in #422 (comment):

I'm grateful to hear that this was a faulty impression of mine.

I wouldn't call it a faulty impression. The project is sending mixed signals.

https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/73 is the most relevant ticket when it comes to producing binaries for additional platforms.

The reality of Forgejo Runner is that it is being used on proprietary and unsupported platforms by some of Forgejo's largest and most prominent users. But I also cannot and do not want to force contributors or maintainers of Forgejo's infrastructure into maintaining or, even worse, owning Windows and macOS machines. Which creates the conundrum we're in.

I dont think this particular combination is possible unless VM support was added for Linux hosts?

https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/107

Regarding requiring Git, separate binaries, built-in actions... you've lost me.

I've outlined the options I can imagine in #422 (comment). There's no progress because each option has significant drawbacks and there is no suitable testing infrastructure.

@stephenwhittle wrote in #422 (comment):

This suggests that even a simple
PUT /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts/{file_name}
would at least have the advantage of being consistent with other new APIs that are provided.

That's more or less what I have in mind. Things that have to be investigated: Do we want to and should we provide the ability to upload arbitrary files, or only archives? If so, what would break? But that discussion should happen in forgejo/forgejo#7926.

@stephenwhittle wrote in #422 (comment):

it would be good to document the existing upload/download endpoints (and any other currently-undocumented endpoints I haven't encountered yet), with parameters and a cURL command like we do for the package API.

I would prefer if we could not do that. If we commit to exposing those APIs, we have to maintain them. I'd rather invest the time in new APIs. Which can be done quickly once we figure out what we want.

@stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14990799: > I'm grateful to hear that this was a faulty impression of mine. I wouldn't call it a faulty impression. The project is sending mixed signals. https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/73 is the most relevant ticket when it comes to producing binaries for additional platforms. The reality of Forgejo Runner is that it is being used on proprietary and unsupported platforms by some of Forgejo's largest and most prominent users. But I also cannot and do not want to force contributors or maintainers of Forgejo's infrastructure into maintaining or, even worse, owning Windows and macOS machines. Which creates the conundrum we're in. > I dont think this particular combination is possible unless VM support was added for Linux hosts? https://code.forgejo.org/forgejo/forgejo-actions-feature-requests/issues/107 Regarding requiring Git, separate binaries, built-in actions... you've lost me. I've outlined the options I can imagine in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-11227879. There's no progress because each option has significant drawbacks and there is no suitable testing infrastructure. @stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14990799: > This suggests that even a simple > `PUT /repos/{owner}/{repo}/actions/runs/{run_id}/artifacts/{file_name}` > would at least have the advantage of being consistent with other new APIs that are provided. That's more or less what I have in mind. Things that have to be investigated: Do we want to and should we provide the ability to upload arbitrary files, or only archives? If so, what would break? But that discussion should happen in https://codeberg.org/forgejo/forgejo/issues/7926. @stephenwhittle wrote in https://codeberg.org/forgejo/discussions/issues/422#issuecomment-14990799: > it would be good to document the existing upload/download endpoints (and any other currently-undocumented endpoints I haven't encountered yet), with parameters and a cURL command like we do for the package API. I would prefer if we could not do that. If we commit to exposing those APIs, we have to maintain them. I'd rather invest the time in new APIs. Which can be done quickly once we figure out what we want.
Sign in to join this conversation.
No Branch/Tag specified
No results found.
No results found.
Labels
Clear labels
User research - Accessibility
Requires input about accessibility features, likely involves user testing.
User research - Blocked
Do not pick as-is! We are happy if you can help, but please coordinate with ongoing redesign in this area.
User research - Community
Community features, such as discovering other people's work or otherwise feeling welcome on a Forgejo instance.
User research - Config (instance)
Instance-wide configuration, authentication and other admin-only needs.
User research - Errors
How to deal with errors in the application and write helpful error messages.
User research - Filters
How filter and search is being worked with.
User research - Future backlog
The issue might be inspiring for future design work.
User research - Git workflow
AGit, fork-based and new Git workflow, PR creation etc
User research - Labels
Active research about Labels
User research - Moderation
Moderation Featuers for Admins are undergoing active User Research
User research - Needs input
Use this label to let the User Research team know their input is requested.
User research - Notifications/Dashboard
Research on how users should know what to do next.
User research - Rendering
Text rendering, markup languages etc
User research - Repo creation
Active research about the New Repo dialog.
User research - Repo units
The repo sections, disabling them and the "Add more" button.
User research - Security
User research - Settings (in-app)
How to structure in-app settings in the future?
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
8 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Reference
forgejo/discussions#422
Reference in a new issue
forgejo/discussions
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?