introduce the concept of adding path dependencies of the build script itself #35473

Followup from #35428

Migrated from https://github.com/ziglang/zig/issues/20605

Related to https://github.com/ziglang/zig/issues/20602

We already have the concept of poison for when it is impossible to cache the
configuration. This issue is for properly tracking when configuration becomes
outdated.

Sometimes the build script should be re-executed. For instance, if the build
script runs git in order to produce a sha1 commit hash that makes it into
configuration options, then that command needs to be re-run sometimes. In this
use case, a file dependency should be added on .git/HEAD in order to track
this. This file dependency is not attached to any build step but rather attached
to running the configure phase (aka build.zig script).

The cases to look out for:

• --watch needs to re-run configuration under these conditions
• zig build CLI needs to account for this in the configuration cache check

Don't forget to implement Cache.addPathPost and
Configuration.Path.toCachePath which are currently @panic("TODO") stubs.

Closure criteria is new std.Build API added for this, standalone test case added,
and use it in zig's own build.zig API for exactly the git use case mentioned above.

Will fix #35460

Another important use case for this feature is scanning directories for test manifests: in the Zig compiler build script, we want to re-run the build script whenever a file is added or removed from test/cases/. (We are also interested in any individual files in that directory changing, since the manifests are parsed at configure time.)

To add to this: In a C project (which uses CMake) of a club I am part of, we observe not just the current git hash but also if the state is dirty or not.
Both of this ends up in a header and becomes later part of the logging output of the final program (in this case the software running on the robots). Both of these pieces of information made debugging across teams quite a bit easier.
It's currently implemented by running a cmake script at every build call (not efficient, but I don't really know how to teach CMake such a dependency).

So, after this here gets implemented, how would one implement this or should one just go and let a small Zig program run unconditionally (by setting has_side_effects of a run step to true) whose output is then the root source file of a separate module?

So, after this here gets implemented, how would one implement this

To be clear, this is already implemented. As of #35428, build.zig logic is cached by default and it is users' responsibility to advertise any additional path dependencies by calling b.dependOnFileContents. The issue closure criteria has not yet been reached however.

As a last resort, the "sledgehammer" option you have available to you is to call b.graph.poisonCache:

Indicates that the configure logic had side effects, or otherwise did something that could not be tracked by the cache system.

When the cache is poisoned, the maker process will delete the build configuration file upon ingesting it since it cannot be reused.

This will cause build scripts to always be rerun when you run zig build.

This would be very bad hygiene for a dependency package to do since it infects the entire build graph, so you can imagine it's especially import to avoid in a reusable package context. For the main application build.zig, it's up to you whether that saves you maintenance burden or causes it.

An alternative you have to this specifically for git dirty state tracking is to add a step to the build graph that fails if it detects dirty state, and then make your publish step depend on that one. This way it doesn't even have to check for dirty state if you're not publishing a release. You could make all development builds "dirty" for example since the programmer did not declare that they were publishing a release.

Another alternative is to make a utility program that computes the version including with git dirty state tracking and use that program's output in the version. This utility program would always need to be rerun with every zig build, but then build scripts could continue to enjoy caching, and if the version string ended up the same, then the application recompilation step could be skipped.

Yeah, this sounds to me like something like this would result in a better state:

build.zig:

conststd=@import("std");pubfnbuild(b:*std.Build)void{constoptimize=b.standardOptimizeOption(.{});consttarget=b.standardTargetOptions(.{});constget_git_info=b.addRunArtifact(b.addExecutable(.{.name="get_git_info",.root_module=b.createModule(.{.root_source_file=b.path("tools/get_git_info.zig"),.optimize=.ReleaseSafe,.target=b.graph.host,}),}));// ensure this will always be runget_git_info.has_side_effects=true;constgit_info=b.createModule(.{// basename is important.root_source_file=get_git_info.captureStdOut(.{.basename="git_info.zig"}),});constprogram=b.createModule(.{.root_source_file=b.path("src/main.zig"),.optimize=optimize,.target=target,});program.addImport("GitInfo",git_info);constexe=b.addExecutable(.{.name="main",.root_module=program,});b.installArtifact(exe);}