roadmap discussion #5

jeffective commented 2025年12月28日 08:16:24 +01:00

@dasimmet, with 8a92c546fd, we can now run an image! (I mean it works on my machine...)

What other features we should have before a 0.1.0 release?

• somehow add an API for downloading an image from a registry to serve as base image?
  • not sure how future proof this will be given zig build system may be sandboxed in future?
  • this is likely to be a mess of https etc. and not sure how credentials for private registries would work either. I don't think people want creds hard-coded into their repo typically. https / crypto will blow up the first-time compilation time too.
• upload to registry? (same concerns here with https etc)
• consider combining our separate exe's into a single exe
  • What advantages / disadvantages does this have?
• polish the APIs a bit more to hopefully capture more use cases, allow arbitrary additional annotations ,etc.
  • annotations
  • compression levels
• test windows support
  • tar produced on windows should be identical to tar produced on linux
• setup at least minimal CI (woodpecker?)
  • zig fmt --ast-check --check .
  • zig build test
  • maybe zig build test-podman
• make the package actually importable as dependency

@dasimmet, with 8a92c546fd, we can now run an image! (I mean it works on my machine...) What other features we should have before a 0.1.0 release? - [ ] somehow add an API for downloading an image from a registry to serve as base image? - not sure how future proof this will be given zig build system may be sandboxed in future? - this is likely to be a mess of https etc. and not sure how credentials for private registries would work either. I don't think people want creds hard-coded into their repo typically. https / crypto will blow up the first-time compilation time too. - [ ] upload to registry? (same concerns here with https etc) - [x] consider combining our separate exe's into a single exe - What advantages / disadvantages does this have? - [ ] polish the APIs a bit more to hopefully capture more use cases, allow arbitrary additional annotations ,etc. - [ ] annotations - [ ] compression levels - [ ] test windows support - tar produced on windows should be identical to tar produced on linux - [x] setup at least minimal CI (woodpecker?) - `zig fmt --ast-check --check .` - `zig build test` - maybe `zig build test-podman` - [x] make the package actually importable as dependency

dasimmet commented 2025年12月29日 16:55:45 +01:00

Cool, yeah i have only tested importing and running images in podman manually so far.

• I think downloading is probably the least priority, since it kind of would require doing that in another run step...and usually zig discurages network access during outside of build.zig.zon. ideally, oci registries would have an endpoint to download a layer that could be added in there, but i doubt that's possible. We could also rely (temporarily) on calling the system container engine (docker/podman, but "skopeo" is best designed for this) to download and export base images as an archive that could serve as a base.

• Upload is kind of the same issue, I'd rather load the output file into a container engine and then push it. skopeo can push a file to a registry directly.

• Single host exe: It saves on individual compilations. All the existing individual commands could just be subcommands selected by argument 1, and the subcommand main() can just receive the IO, allocator and remaining arguments as parameters.

• polish the apis / import as dependency: I've been working on a POC for usage from dependency it here: https://codeberg.org/jeffective/oci/src/branch/feature/build-interface/example/build.zig but its quite a big change and i had to comment all the build.zig code temporarily. Try it by running "zig build" from the examples subdirectory.
  I'm not a big fan of using WriteFiles for our use case, because it creates unneccessary copies of files in the build cache.
  Similar to how zig's internal build functions work (Compile Step -> modules), I think its easier to accomplish a cleaner and more flexible build.zig api by creating images top-down, and by not starting with the layer but the index, then add a manifest to the index, add the layer to the manifest and the files to the layer.
  For convenience, the "CreateOptions" can of course also provide a way to create everything in one go.

• Something that we may need to pay attention to is blob deduplication. On the layer creation side, the build system will figure it out, but create-index only needs to add them once to the final tar archive.

• Windows support: for that the main issue might be filesystem permissions and ownership. At the moment we do not pass those to create_layer, but for example to mark executables executable it is required. On unix build hosts we could just use host permissions but I assume for it to be the same binary on windows it needs to be set through the build system instead. Which is more challenging when dealing with directories (maybe one setting for all subdirs and one for all subfiles would be enough).

• ci: sure, but I have no experience with Woodpecker or codeberg, mostly only gitlab. Maybe i can hook up my VPS as a runner if need be.

Cool, yeah i have only tested importing and running images in podman manually so far. - I think downloading is probably the least priority, since it kind of would require doing that in another run step...and usually zig discurages network access during outside of build.zig.zon. ideally, oci registries would have an endpoint to download a layer that could be added in there, but i doubt that's possible. We could also rely (temporarily) on calling the system container engine (docker/podman, but "skopeo" is best designed for this) to download and export base images as an archive that could serve as a base. - Upload is kind of the same issue, I'd rather load the output file into a container engine and then push it. skopeo can push a file to a registry directly. - Single host exe: It saves on individual compilations. All the existing individual commands could just be subcommands selected by argument 1, and the subcommand main() can just receive the IO, allocator and remaining arguments as parameters. - polish the apis / import as dependency: I've been working on a POC for usage from dependency it here: <https://codeberg.org/jeffective/oci/src/branch/feature/build-interface/example/build.zig> but its quite a big change and i had to comment all the build.zig code temporarily. Try it by running "zig build" from the examples subdirectory. I'm not a big fan of using WriteFiles for our use case, because it creates unneccessary copies of files in the build cache. Similar to how zig's internal build functions work (Compile Step -> modules), I think its easier to accomplish a cleaner and more flexible build.zig api by creating images top-down, and by not starting with the layer but the index, then add a manifest to the index, add the layer to the manifest and the files to the layer. For convenience, the "CreateOptions" can of course also provide a way to create everything in one go. - Something that we may need to pay attention to is blob deduplication. On the layer creation side, the build system will figure it out, but create-index only needs to add them once to the final tar archive. - Windows support: for that the main issue might be filesystem permissions and ownership. At the moment we do not pass those to create_layer, but for example to mark executables executable it is required. On unix build hosts we could just use host permissions but I assume for it to be the same binary on windows it needs to be set through the build system instead. Which is more challenging when dealing with directories (maybe one setting for all subdirs and one for all subfiles would be enough). - ci: sure, but I have no experience with Woodpecker or codeberg, mostly only gitlab. Maybe i can hook up my VPS as a runner if need be.

dasimmet commented 2026年01月09日 18:58:56 +01:00

Ive done some testing with skopeo, and it looks like in the current state multiple manifests are only recognized as different images, not as a single multi-platform one.
When copying a multi-platform docker image to an oci archive or directory, the root index.json only points to another single blob of type "index", that in turn points to the individual Manifests. I guess our api should support both cases.

Ive done some testing with skopeo, and it looks like in the current state multiple manifests are only recognized as different images, not as a single multi-platform one. When copying a multi-platform docker image to an oci archive or directory, the root index.json only points to another single blob of type "index", that in turn points to the individual Manifests. I guess our api should support both cases.

dasimmet commented 2026年02月10日 09:02:25 +01:00

For m, project i have a working multiarch build using the post-juicymain branch:

https://gitlab.com/dasimmet/zig-mirror/-/pipelines/2313112307

I add a resolv.conf and ssl cert directory from another dependency and use skopeo copy to upload in a separate job. We could integrate skopeo as a system command, but i dont have it in the same container as zig anyways.

However i sometimes get a ParseResolvconfFailed error at runtime (might be a zig 16 issue)

Should i merge the zig upgrade? It definately breaks projects before the file io changes.

For m, project i have a working multiarch build using the post-juicymain branch: https://gitlab.com/dasimmet/zig-mirror/-/pipelines/2313112307 I add a resolv.conf and ssl cert directory from another dependency and use skopeo copy to upload in a separate job. We could integrate skopeo as a system command, but i dont have it in the same container as zig anyways. However i sometimes get a ParseResolvconfFailed error at runtime (might be a zig 16 issue) Should i merge the zig upgrade? It definately breaks projects before the file io changes.

jeffective commented 2026年02月10日 21:56:51 +01:00

Awesome! Sorry I haven't put much work into this project lately, been distracted.

Feel free to upgrade the zig version as often as possible. I think we should hold at 0.16 when it is released though.

Awesome! Sorry I haven't put much work into this project lately, been distracted. Feel free to upgrade the zig version as often as possible. I think we should hold at 0.16 when it is released though.

jeffective commented 2026年02月10日 22:00:03 +01:00

I would really like to get to a point where we can upload without a system command. I want to be able to ship container applications with zig as the only dependency. If the easiest way to get to that point is to build a c project with zig, then sure. If we can get a minimal implementation using std.net or http that would be awesome.

I would really like to get to a point where we can upload without a system command. I want to be able to ship container applications with zig as the only dependency. If the easiest way to get to that point is to build a c project with zig, then sure. If we can get a minimal implementation using std.net or http that would be awesome.

jeffective commented 2026年03月27日 06:50:49 +01:00

@dasimmet thanks for making progress on this project, I havent forgotten about it, I anticipate another burst of work from me around 0.16, likely when I finally get my primary project to compile again after so many std lib breaking changes, youre awesome!
Its a goal of mine to stop using system installed docker to build that project :)

@dasimmet thanks for making progress on this project, I havent forgotten about it, I anticipate another burst of work from me around 0.16, likely when I finally get my [primary project](https://codeberg.org/jeffective/gatorcat) to compile again after so many std lib breaking changes, youre awesome! Its a goal of mine to stop using system installed docker to build that project :)

dasimmet commented 2026年03月27日 09:53:03 +01:00

No problem...
I think at the top level, using the oci directory format is handier than an archive, parsing blobs requires "random access",
especially as the output of a "FROM" step.

I looked into pulling and pushing (and on a locally running registry pushing works), but at docker.io
or gitlab's registry authentication is pretty tricky (even public images always need bearer tokens) and i get strange errors
that may have to do with zig's http stack...

No problem... I think at the top level, using the oci directory format is handier than an archive, parsing blobs requires "random access", especially as the output of a "FROM" step. I looked into pulling and pushing (and on a locally running registry pushing works), but at docker.io or gitlab's registry authentication is pretty tricky (even public images always need bearer tokens) and i get strange errors that may have to do with zig's http stack...

jeffective commented 2026年04月28日 10:10:21 +02:00

@dasimmet, I added some rudimentary CI. its based on alpine container so it might be hard to get a container runtime in there but at least its something

@dasimmet, I added some rudimentary CI. its based on alpine container so it might be hard to get a container runtime in there but at least its something

dasimmet commented 2026年04月29日 18:57:27 +02:00

@jeffective wrote in #5 (comment):

@dasimmet, I added some rudimentary CI. its based on alpine container so it might be hard to get a container runtime in there but at least its something

pretty cool ;-D is it running on mrs....

i've been working on the FROM support on a branch....

@jeffective wrote in https://codeberg.org/jeffective/oci/issues/5#issuecomment-13922351: > @dasimmet, I added some rudimentary CI. its based on alpine container so it might be hard to get a container runtime in there but at least its something pretty cool ;-D is it running on mrs.... i've been working on the FROM support on a branch....

jeffective commented 2026年05月08日 07:45:45 +02:00

This is AMAZING!

info: manifest url: http://localhost:5000/v2/gatorcat/manifests/0.4.10
info: payload:
{
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.index.v1+json",
 "manifests": [
 {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:e01db601643b91be9be39e76dc29d763ed3729c9f5757402673d4631a1c860fe",
 "size": 407,
 "annotations": {
 "org.opencontainers.image.ref.name": "localhost:5000/gatorcat:0.4.10"
 },
 "platform": {
 "architecture": "amd64",
 "os": "linux"
 }
 },
 {
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "digest": "sha256:9fb3586deb2fbc8b9d0a7c65ac101961ccceef06e150ca34a19f774b521c1f8f",
 "size": 406,
 "annotations": {
 "org.opencontainers.image.ref.name": "localhost:5000/gatorcat:0.4.10"
 },
 "platform": {
 "architecture": "arm64",
 "os": "linux"
 }
 }
 ]
}
info: response status: .created

$ docker run localhost:5000/gatorcat:0.4.10 version
0.4.10

This is AMAZING! ``` info: manifest url: http://localhost:5000/v2/gatorcat/manifests/0.4.10 info: payload: { "schemaVersion": 2, "mediaType": "application/vnd.oci.image.index.v1+json", "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:e01db601643b91be9be39e76dc29d763ed3729c9f5757402673d4631a1c860fe", "size": 407, "annotations": { "org.opencontainers.image.ref.name": "localhost:5000/gatorcat:0.4.10" }, "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.oci.image.manifest.v1+json", "digest": "sha256:9fb3586deb2fbc8b9d0a7c65ac101961ccceef06e150ca34a19f774b521c1f8f", "size": 406, "annotations": { "org.opencontainers.image.ref.name": "localhost:5000/gatorcat:0.4.10" }, "platform": { "architecture": "arm64", "os": "linux" } } ] } info: response status: .created ``` ```sh $ docker run localhost:5000/gatorcat:0.4.10 version 0.4.10 ```

jeffective commented 2026年05月08日 08:49:59 +02:00

jeffective/gatorcat@3510a7dccf
image

https://codeberg.org/jeffective/gatorcat/commit/3510a7dccf8b74b6cc8b06b4d0c1ca72e94db4c1