fetch packages into project-local directory #31121

closes https://github.com/ziglang/zig/issues/14283

Release Notes

Instead of being fetched into $GLOBAL_ZIG_CACHE/p/$HASH, package dependencies are now fetched into a "zig-pkg" directory relative to the build root (next to build.zig). Users are generally encouraged to not commit these files to source control, however it is understood that some will choose to do so for convenience.

After a package is fetched, the filters are applied (paths field in build.zig.zon) in order to delete files not part of the hash, and then the package is recompressed into a canonical $GLOBAL_ZIG_CACHE/p/$HASH.tar.gz in order to avoid network next time the same package is needed.

The motivation for this change is to make it easier to tinker. Go ahead and edit those files, see what happens. Swap out your package directory with a git clone. Grep your dependencies all together. Configure your IDE to auto-complete based on zig-pkgs directory. Run baobab on your dependency tree. Furthermore, by having the global cache have compressed files instead makes it easier to share that cached data between computers.

zig build will now fail when encountering package dependencies without fingerprint field or with name as a string rather than enum literal. Fingerprint is needed in order to determine that two packages with different versions are intended to be different versions of the same project. It will become an error to have the same fingerprint, same version, different hash in your dependency tree because it means somebody forgot to bump a version number, or somebody is trying to do a hostile package fork and now you have to choose a side.

Zig no longer observes ZIG_BTRFS_WORKAROUND environment variable. The bug has been fixed in upstream Linux a long time ago by now. https://github.com/ziglang/zig/issues/17095

Followup Work

• #31139
• when showing package fetching errors, also show dependency trace (which package drags in the package with the error?)
• ability to specify name, version, and paths for dependency URLs that do not have a build.zig.zon file inside
• #31124
• https://github.com/ziglang/zig/issues/14288
• https://github.com/ziglang/zig/issues/23236
• a tool for verifying integrity of package directories (detects when the hash does not match the contents)

Will this zig-pkg directory still be present when you are referencing relative file system paths to dependencies? Not just https/git urls?

Package dependencies are now fetched into a "zig-pkg" directory relative to the build root (next to build.zig). Users are generally encouraged to not commit these files to source control, however it is understood that some will choose to do so for convenience.

Could you better express when committing this dir could be sensible?

Say my use-case is that I want to ensure that anybody working with my repo has the full source code of the project (including deps) after cloning it the first time. Would intentionally committing the zig-pkg directory be a supported way to achieve this?

Example: A package manager might want to be able to build everything from source in an offline container, and having the dependencies fetched upfront helps. Another reason might be a concern that some dependencies that the project depends on might be taken down (or face network issues) and it would just be nice to keep a local copy.

This can currently be achieved by picking a directory to copy the dependencies to, and using an explicit path to refer to said dependency. But it doesn't work for nested dependencies (dependencies used by dependencies of the current project) and there's also no idiomatic way to find out what version of the deps that the project is depending on.

Sorry if what I outlined above would better be discussed in a different issue.

Will this zig-pkg directory still be present when you are referencing relative file system paths to dependencies? Not just https/git urls?

It won't be created for dependencies that use relative paths, only for fetched packages

Instead of being fetched into $GLOBAL_ZIG_CACHE/p/$HASH, package dependencies are now fetched into a "zig-pkg" directory relative to the build root (next to build.zig). Users are generally encouraged to not commit these files to source control, however it is understood that some will choose to do so for convenience.

This is the user-facing change: packages will now be stored in ./zig-pkg/pkg-hash relative to the build.zig of your project

After a package is fetched, the filters are applied (paths field in build.zig.zon) in order to delete files not part of the hash, and then the package is recompressed into a canonical $GLOBAL_ZIG_CACHE/p/$HASH.tar.gz in order to avoid network next time the same package is needed.

"delete files" gave me a scare, but this is already the case:
newly downloaded packages are placed into a temporary directory and pruned.
No changes - this is just an implementation details description.

The motivation for this change is to make it easier to tinker. Go ahead and edit those files, see what happens. Swap out your package directory with a git clone. Grep your dependencies all together. Configure your IDE to auto-complete based on zig-pkgs directory. Run baobab on your dependency tree. Furthermore, by having the global cache have compressed files instead makes it easier to share that cached data between computers.

"tinker" is the right term here - browse the codebase, maybe fix a compile error if using a newer nightly that the required. If serious about making any changes one should be using clones and relative paths

Fair and Good:

• Grep and IDE assisted, no language servers, auto-complete
• Solves the uses case presented by sagehane

Con:

• multiple copies of the same package

Edit: Corrections. Just ran it and I was wrong about a few things, apologies.

Will it be possible to prefetch dependecies before running 'zig build'?

Will it be possible to prefetch dependecies before running 'zig build'?

$ zig build --help
...
 --fetch[=mode] Fetch dependency tree (optionally choose laziness) and exit
 needed (Default) Lazy dependencies are fetched as needed
 all Lazy dependencies are always fetched
...

Available since zig 0.12.0. https://github.com/ziglang/zig/issues/14280

Is there a consideration of falling back to the dep name in build.zig.zon when the packing being used doesn't have a build.zig.zon file? For example when fetching C or other non-standard Zig package.

Currently the path name being used in the following format:

zig-pkg/N-V-__8AANlkNAB77Z946lqp-lgp2qSsOBDADValZC86PhB-

I wonder if it makes sense to do likewise for std? That is, right now I get an error that looks like this:

/Users/matklad/.cache/zig/p/N-V-__8AALEiFxbnKVxP_rPHB849sj7v5YW_fKde4fcxqcBZ/lib/std/debug.zig:420:14: 0x1044e8043 in assert (cst)
 if (!ok) unreachable; // assertion failure
 ^
/Users/matklad/p/csc/csc.zig:112:19: 0x1045edf1b in monotonicity (cst)
 assert(old.header[old_index].view == new.header[new_index].view);

Would be cool if it looked like this instead:

/Users/matklad/p/csc/zig-pkg/std/debug.zig:420:14: 0x1044e8043 in assert (cst)
 if (!ok) unreachable; // assertion failure
 ^
/Users/matklad/p/csc/csc.zig:112:19: 0x1045edf1b in monotonicity (cst)
 assert(old.header[old_index].view == new.header[new_index].view);

I think this will be a tremendous help for people learning to program, having user-friendly access to std like this right from your project's directory, and ability to just go and add debug prints in the middle of std code. Basically, the same tinkering argument.

On the other hand, I would have 17mb of std copied into a large number of projects and that's a lot of wasted disk space for my use case.