zig build: separate the maker process from the configurer process #35428

andrewrk commented 2026年05月23日 10:01:18 +02:00

closes #31397
closes #31691

Release Notes

The primary motivation is to make zig build faster, both in the sense that it will take less time to compile, because only the user's build.zig logic is being recompiled when it changes, and in the sense that the build runner is built with optimizations enabled, which is starting to become more valuable now that we have introduced --watch and --fuzz. Furthermore, the build.zig logic can be skipped sometimes depending on what CLI flags are used with zig build.

This changeset heavily reworks the internal mechanism of the zig build system, however, it is mostly non-breaking from an API perspective, with the exceptions noted below.

Third-party tooling such as ZLS will benefit from consuming the serialized configuration file rather than maintaining a fork of the build runner.

New CLI Arguments

 --maker-opt=[mode] Change maker executable optimization mode (default: ReleaseSafe)

--print-configuration Render configuration as .zon to stdout

Run Step: Passthru Args

In the Run step, passthru args are all together now, not observable in
configure phase whether run args are provided.

if(b.args)|args|{run_cmd.addArgs(args);}

⬇️

run_cmd.addPassthruArgs();

This removes a capability from build scripts since they can no longer observe
those arguments. In exchange, it means that when changing those arguments,
build scripts no longer must be rebuilt from source.

Fmt Step: Options

paths and exclude_paths are now LazyPath lists. There is a convenience method to create them: b.pathList.

- const fmt_include_paths = &.{ "lib", "src", "test", "tools", "build.zig", "build.zig.zon" };
- const fmt_exclude_paths = &.{ "test/cases", "test/behavior/zon" };
+ const fmt_include_paths = b.pathList(&.{ "lib", "src", "test", "tools", "build.zig", "build.zig.zon" });
+ const fmt_exclude_paths = b.pathList(&.{ "test/cases", "test/behavior/zon" });

std.Build API

• b.build_root (Directory) -> b.root (Path)
• ConfigHeader.Options: include_guard_override -> include_guard
• LazyPath: getDisplayName -> format ("{f}")
• LazyPath.basename: removed since the value is not known until make phase
• b.findProgram divided into findProgram and findProgramLazy and API future-proofed.
• ConfigHeader now properly reports unused values for all styles

std.Target API

• parseCpuModel now returns optional rather than error

Build System Behavior

• When targeting Windows or Wine, artifact args added to Run steps used to modify PATH based on the set of directories containing the recursive set of DLL dependencies. Now this is only done for argv[0]. The motivation for also doing this for the other command line arguments is unclear, since those DLLs don't need to be loaded in order to execute argv[0].

Perf Data Point: zig build -h (cached)

Benchmark 1 (34 runs): master/zig build -h
 measurement mean ± σ min ... max outliers delta
 wall_time 150ms ± 5.52ms 145ms ... 165ms 4 (12%) 0%
 peak_rss 84.8MB ± 275KB 84.2MB ... 85.1MB 0 ( 0%) 0%
 cpu_cycles 593M ± 4.01M 588M ... 608M 2 ( 6%) 0%
 instructions 995M ± 52.5K 995M ... 995M 0 ( 0%) 0%
 cache_references 25.8M ± 165K 25.4M ... 26.1M 0 ( 0%) 0%
 cache_misses 651K ± 20.1K 619K ... 697K 0 ( 0%) 0%
 branch_misses 918K ± 7.44K 906K ... 935K 0 ( 0%) 0%
Benchmark 2 (348 runs): branch/zig build -h
 measurement mean ± σ min ... max outliers delta
 wall_time 14.3ms ± 744us 13.2ms ... 23.3ms 8 ( 2%) ⚡- 90.4% ± 0.4%
 peak_rss 78.5MB ± 562KB 77.1MB ... 81.4MB 7 ( 2%) ⚡- 7.4% ± 0.2%
 cpu_cycles 24.1M ± 821K 22.8M ... 27.1M 3 ( 1%) ⚡- 95.9% ± 0.1%
 instructions 43.7M ± 23.8K 43.7M ... 43.8M 56 (16%) ⚡- 95.6% ± 0.0%
 cache_references 1.46M ± 14.6K 1.40M ... 1.50M 19 ( 5%) ⚡- 94.3% ± 0.1%
 cache_misses 142K ± 4.87K 127K ... 157K 2 ( 1%) ⚡- 78.1% ± 0.4%
 branch_misses 126K ± 1.37K 120K ... 129K 12 ( 3%) ⚡- 86.3% ± 0.1%

Perf Data Point: Running a subset of the toolchain test suite

Fresh cache except in both cases I let the build runner / maker / configurer
get cached beforehand.

master:

$ time -v stage3/bin/zig build test-cases test-standalone -Dskip-release --seed 0
 Elapsed (wall clock) time (h:mm:ss or m:ss): 2:09.93
 Maximum resident set size (kbytes): 496512

branch:

$ time -v stage3/bin/zig build test-cases test-standalone -Dskip-release --seed 0 --maker-opt=ReleaseFast
 Elapsed (wall clock) time (h:mm:ss or m:ss): 1:46.88
 Maximum resident set size (kbytes): 495700

It's only one data point, but the new thing is faster and less memory. As is
expected from being compiled in optimized mode. That's the point of the
changeset.

Introduce the Concept of Configure Cache Poisoning

If the cache is poisoned means that the configure logic had side
effects, or otherwise did something that could not be tracked by the
cache system.

This is not to be confused with whether individual steps may have side
effects when being evaluated; it has to do with the logic inside build.zig
itself. For example, a Run step that prints "hello world" has side
effects at make time and therefore does not warrant setting this flag,
while checking for the existence of scdoc at configure time in order to
choose the default value for a configuration option does.

Keeping the cache pure will make zig build faster, bypassing the
configurer process when identical configuration would be generated.

When the cache is poisoned, the maker process will delete the build
configuration file upon ingesting it since it cannot be reused.

Currently the only way to poison the cache is to call findProgram.

Advanced users can override the cache poisoning behavior with a CLI option:

 --cache-poison[=mode] Override configuration caching behavior
 pure (default) Avoid false positive cache hits
 poisoned Don't cache the configuration
 disallowed Panics when cache would be poisoned
 ignored A little poison never hurt anybody

findProgram

Immediately (in the configure phase), searches for an executable on the host
that has more than one possible name.

Names are searched in order, observing search prefixes first and then PATH
environment variable.

Calling this function poisons the configuration cache, so it is only
appropriate when the existence of the program or its output needs to be
observed by configuration logic. That's why there is a lazy variant of this function now.

findProgramLazy

Creates an anonymous Step that searches for an executable on the host that
has more than one possible name.

Returns the LazyPath of the found executable. The search only takes place
if the LazyPath will be used by a depending Step.

This API is useful in the following cases:

• The binary is not named the same across all systems (for example "python"
  vs "python3").
• The binary may be produced by building from source rather than being
  globally installed and will therefore be possibly found in one of the
  search prefix paths.

Removed Ability to Override Build Runner

There is no concept of a "build runner" any more; it has been split in two:
configurer and maker.

Users of this feature will likely want to no longer override any logic, and
instead consume the configuration file produced by configurer.

If overriding one or the other of these is desired, the feature will need to be
re-introduced (as --override-maker or --override-configurer).

Followup Issues

• CLI flag to print path to the configuration file
• test a bunch of third party projects / help people migrate
• search prefixes added at configuration time should be package-local
• introduce the concept of adding path dependencies of the configure script itself
  • don't forget to implement Cache.addPathPost and Configuration.Path.toCachePath
  • make --watch rerun itself
• Maker.WebServer: fix bad logic in serveTarFile
• Maker.WebServer: don't do linear search in the steps array
• maker: refactor and reuse the fuzz number parsing here (--maxrss, --debounce)
• handle missing cache hits when chaining two run steps
• Absolute and cwd-relative paths in build cache
• build system fmt step with check=false does not acquire a write lock on source files
• enhance CheckFile step output when there is not a match
• stop leaking into global process arena. audit all uses of graph.arena
• reduce the size of Maker.Step.Extended (make Run smaller) probably by using an arena per make
• consider using ReleaseFast by default for Maker
• make the generated dependencies.zig be dependencies.zon and don't put absolute paths in there
  • and adjust dependencyInner to not openDir()
• make more stuff use IndexType
• make addExtra return Index using reflection
• refactor with DefaultingEnum
• get the target from the parent process instead
• link_eh_frame_hdr should be DefaultingBool
• make --foo, --no-foo CLI args uniform (make them -f args instead)
• install steps should provide generated files for installed things, then delete the run step hack
  • but artifact install steps also add paths for dyn libs on windows
• no more "artifact arg" to run step. if you want to run the post-install binary, get the lazy path
  from the install step.
• UpdateSourceFiles: introduce Group
• WriteFiles: introduce Group
• re-examine the use case of adding file paths to Options steps
• extract the reusable Configure abstractions and reuse it for Zir etc
• add the ability to delete files to UpdateSourceFiles
• merge the code of Maker.Step.Compile.checkCompileErrors with std.testing.expectEqualStrings
• std.Build.Run: make addPathDir use LazyPath. This will require enhancing the env_map to support
  some values being strings and some values being LazyPaths.
  • also this is not the appropriate place to integrate with wine. move it to Maker logic
• constrain the set of environment variables passed to configurer. they can ask for more but they
  will integrate properly with the cache system.
• should ConfigHeader run in configurer and store the rendered string?
• ConfigHeader:
  • TODO: use C-specific escaping instead of zig string literals
  • TODO: use nasm-specific escaping instead of zig string literals
• enhance doctest to use "--listen=-" rather than operating in a temporary directory
• Maker: eliminate memory allocation in the hot path (search for "TODO eliminate memory allocation here" x2)
• print more stuff in --print-configuration
  • paths
  • unlazy deps
  • system integrations
  • available options
• Maker.Step.InstallDir: update the call to truncatePath to first check dest
  stat. if already exists and length zero, skip the open() call and track the
  "all_cached" flag properly like the surrounding code.
• Maker.Step.Run: when trying to interpret, learn the target from the binary
  directly rather than from relying on it being a Compile step. This will make
  this logic work even for the edge case that the binary was produced by a
  third party.
• investigate why configurer takes so long to compile
• fmt step: import zig fmt code directly rather than child proc?
• add zig reduce functionality directly into the build system. detect when the compiler crashes and
  offer a command that will attempt to make a reproducer.

closes #31397 closes #31691 ## Release Notes The primary motivation is to make `zig build` faster, both in the sense that it will take less time to compile, because only the user's `build.zig` logic is being recompiled when it changes, and in the sense that the build runner is built with optimizations enabled, which is starting to become more valuable now that we have introduced `--watch` and `--fuzz`. Furthermore, the build.zig logic can be skipped sometimes depending on what CLI flags are used with `zig build`. This changeset heavily reworks the internal mechanism of the zig build system, however, it is mostly non-breaking from an API perspective, with the exceptions noted below. Third-party tooling such as ZLS will benefit from consuming the serialized configuration file rather than maintaining a fork of the build runner. ### New CLI Arguments ``` --maker-opt=[mode] Change maker executable optimization mode (default: ReleaseSafe) ``` ``` --print-configuration Render configuration as .zon to stdout ``` ### Run Step: Passthru Args In the Run step, passthru args are all together now, not observable in configure phase whether run args are provided. ```zig if (b.args) |args| { run_cmd.addArgs(args); } ``` ⬇️ ```zig run_cmd.addPassthruArgs(); ``` This removes a capability from build scripts since they can no longer observe those arguments. In exchange, it means that when changing those arguments, build scripts no longer must be rebuilt from source. ### Fmt Step: Options `paths` and `exclude_paths` are now LazyPath lists. There is a convenience method to create them: `b.pathList`. ```diff - const fmt_include_paths = &.{ "lib", "src", "test", "tools", "build.zig", "build.zig.zon" }; - const fmt_exclude_paths = &.{ "test/cases", "test/behavior/zon" }; + const fmt_include_paths = b.pathList(&.{ "lib", "src", "test", "tools", "build.zig", "build.zig.zon" }); + const fmt_exclude_paths = b.pathList(&.{ "test/cases", "test/behavior/zon" }); ``` ### std.Build API * `b.build_root` (Directory) -> `b.root` (Path) * `ConfigHeader.Options`: `include_guard_override` -> `include_guard` * `LazyPath`: `getDisplayName` -> `format` ("{f}") * `LazyPath.basename`: removed since the value is not known until make phase * `b.findProgram` divided into `findProgram` and `findProgramLazy` and API future-proofed. * `ConfigHeader` now properly reports unused values for all styles ### std.Target API * parseCpuModel now returns optional rather than error ### Build System Behavior * When targeting Windows or Wine, artifact args added to Run steps used to modify PATH based on the set of directories containing the recursive set of DLL dependencies. Now this is only done for argv[0]. The motivation for also doing this for the other command line arguments is unclear, since those DLLs don't need to be loaded in order to execute argv[0]. ### Perf Data Point: `zig build -h` (cached) ``` Benchmark 1 (34 runs): master/zig build -h measurement mean ± σ min ... max outliers delta wall_time 150ms ± 5.52ms 145ms ... 165ms 4 (12%) 0% peak_rss 84.8MB ± 275KB 84.2MB ... 85.1MB 0 ( 0%) 0% cpu_cycles 593M ± 4.01M 588M ... 608M 2 ( 6%) 0% instructions 995M ± 52.5K 995M ... 995M 0 ( 0%) 0% cache_references 25.8M ± 165K 25.4M ... 26.1M 0 ( 0%) 0% cache_misses 651K ± 20.1K 619K ... 697K 0 ( 0%) 0% branch_misses 918K ± 7.44K 906K ... 935K 0 ( 0%) 0% Benchmark 2 (348 runs): branch/zig build -h measurement mean ± σ min ... max outliers delta wall_time 14.3ms ± 744us 13.2ms ... 23.3ms 8 ( 2%) ⚡- 90.4% ± 0.4% peak_rss 78.5MB ± 562KB 77.1MB ... 81.4MB 7 ( 2%) ⚡- 7.4% ± 0.2% cpu_cycles 24.1M ± 821K 22.8M ... 27.1M 3 ( 1%) ⚡- 95.9% ± 0.1% instructions 43.7M ± 23.8K 43.7M ... 43.8M 56 (16%) ⚡- 95.6% ± 0.0% cache_references 1.46M ± 14.6K 1.40M ... 1.50M 19 ( 5%) ⚡- 94.3% ± 0.1% cache_misses 142K ± 4.87K 127K ... 157K 2 ( 1%) ⚡- 78.1% ± 0.4% branch_misses 126K ± 1.37K 120K ... 129K 12 ( 3%) ⚡- 86.3% ± 0.1% ``` ### Perf Data Point: Running a subset of the toolchain test suite Fresh cache except in both cases I let the build runner / maker / configurer get cached beforehand. master: ``` $ time -v stage3/bin/zig build test-cases test-standalone -Dskip-release --seed 0 Elapsed (wall clock) time (h:mm:ss or m:ss): 2:09.93 Maximum resident set size (kbytes): 496512 ``` branch: ``` $ time -v stage3/bin/zig build test-cases test-standalone -Dskip-release --seed 0 --maker-opt=ReleaseFast Elapsed (wall clock) time (h:mm:ss or m:ss): 1:46.88 Maximum resident set size (kbytes): 495700 ``` It's only one data point, but the new thing is faster and less memory. As is expected from being compiled in optimized mode. That's the point of the changeset. ### Introduce the Concept of Configure Cache Poisoning If the cache is poisoned means that the **configure logic** had side effects, or otherwise did something that could not be tracked by the cache system. This is not to be confused with whether individual steps may have side effects when being evaluated; it has to do with the logic inside build.zig itself. For example, a `Run` step that prints "hello world" has side effects *at make time* and therefore does not warrant setting this flag, while checking for the existence of `scdoc` *at configure time* in order to choose the default value for a configuration option does. Keeping the cache pure will make `zig build` faster, bypassing the configurer process when identical configuration would be generated. When the cache is poisoned, the maker process will delete the build configuration file upon ingesting it since it cannot be reused. Currently the only way to poison the cache is to call `findProgram`. Advanced users can override the cache poisoning behavior with a CLI option: ``` --cache-poison[=mode] Override configuration caching behavior pure (default) Avoid false positive cache hits poisoned Don't cache the configuration disallowed Panics when cache would be poisoned ignored A little poison never hurt anybody ``` #### findProgram Immediately (in the configure phase), searches for an executable on the host that has more than one possible name. Names are searched in order, observing search prefixes first and then PATH environment variable. Calling this function poisons the configuration cache, so it is only appropriate when the existence of the program or its output needs to be observed by configuration logic. That's why there is a lazy variant of this function now. #### findProgramLazy Creates an anonymous `Step` that searches for an executable on the host that has more than one possible name. Returns the `LazyPath` of the found executable. The search only takes place if the `LazyPath` will be used by a depending `Step`. This API is useful in the following cases: * The binary is not named the same across all systems (for example "python" vs "python3"). * The binary may be produced by building from source rather than being globally installed and will therefore be possibly found in one of the search prefix paths. ### Removed Ability to Override Build Runner There is no concept of a "build runner" any more; it has been split in two: configurer and maker. Users of this feature will likely want to no longer override any logic, and instead consume the configuration file produced by configurer. If overriding one or the other of these is desired, the feature will need to be re-introduced (as --override-maker or --override-configurer). ## Followup Issues * [CLI flag to print path to the configuration file](https://codeberg.org/ziglang/zig/issues/35498) * test a bunch of third party projects / help people migrate * [search prefixes added at configuration time should be package-local](https://codeberg.org/ziglang/zig/issues/35472) * [introduce the concept of adding path dependencies of the configure script itself](https://codeberg.org/ziglang/zig/issues/35473) - don't forget to implement Cache.addPathPost and Configuration.Path.toCachePath - [make --watch rerun itself](https://github.com/ziglang/zig/issues/20602) * [Maker.WebServer: fix bad logic in serveTarFile](https://codeberg.org/ziglang/zig/issues/35457) * [Maker.WebServer: don't do linear search in the steps array](https://codeberg.org/ziglang/zig/issues/35458) * [maker: refactor and reuse the fuzz number parsing here (--maxrss, --debounce)](https://codeberg.org/ziglang/zig/issues/35459) * [handle missing cache hits when chaining two run steps](https://codeberg.org/ziglang/zig/pulls/30762) * [Absolute and cwd-relative paths in build cache](https://codeberg.org/ziglang/zig/issues/32097) * [build system fmt step with check=false does not acquire a write lock on source files](https://codeberg.org/ziglang/zig/issues/35204) * [enhance CheckFile step output when there is not a match](https://codeberg.org/ziglang/zig/issues/35208) * [stop leaking into global process arena. audit all uses of graph.arena](https://github.com/ziglang/zig/issues/20603) * reduce the size of Maker.Step.Extended (make Run smaller) probably by using an arena per make * consider using ReleaseFast by default for Maker * make the generated dependencies.zig be dependencies.zon and don't put absolute paths in there - and adjust dependencyInner to not openDir() * make more stuff use IndexType * make addExtra return Index using reflection * refactor with DefaultingEnum * get the target from the parent process instead * link_eh_frame_hdr should be DefaultingBool * make --foo, --no-foo CLI args uniform (make them -f args instead) * install steps should provide generated files for installed things, then delete the run step hack - but artifact install steps also add paths for dyn libs on windows * no more "artifact arg" to run step. if you want to run the post-install binary, get the lazy path from the install step. * UpdateSourceFiles: introduce Group * WriteFiles: introduce Group * re-examine the use case of adding file paths to Options steps * extract the reusable Configure abstractions and reuse it for Zir etc * add the ability to delete files to UpdateSourceFiles * merge the code of Maker.Step.Compile.checkCompileErrors with std.testing.expectEqualStrings * std.Build.Run: make addPathDir use LazyPath. This will require enhancing the env_map to support some values being strings and some values being LazyPaths. - also this is not the appropriate place to integrate with wine. move it to Maker logic * constrain the set of environment variables passed to configurer. they can ask for more but they will integrate properly with the cache system. * should ConfigHeader run in configurer and store the rendered string? * ConfigHeader: - TODO: use C-specific escaping instead of zig string literals - TODO: use nasm-specific escaping instead of zig string literals * enhance doctest to use "--listen=-" rather than operating in a temporary directory * Maker: eliminate memory allocation in the hot path (search for "TODO eliminate memory allocation here" x2) * print more stuff in --print-configuration - paths - unlazy deps - system integrations - available options * Maker.Step.InstallDir: update the call to truncatePath to first check dest stat. if already exists and length zero, skip the open() call and track the "all_cached" flag properly like the surrounding code. * Maker.Step.Run: when trying to interpret, learn the target from the binary directly rather than from relying on it being a Compile step. This will make this logic work even for the edge case that the binary was produced by a third party. * investigate why configurer takes so long to compile * fmt step: import zig fmt code directly rather than child proc? * add zig reduce functionality directly into the build system. detect when the compiler crashes and offer a command that will attempt to make a reproducer.

bfredl commented 2026年05月23日 13:14:34 +02:00

Is there a long-term reason that that Maker needs to be compiled at first use, and not simply being built in like zig maker or whatever? (I understand that it is much easier to hack on it during development this way!). As it lives in global cache and not per-project cache, it doesn't depend on any project specific options, no?

Is there a long-term reason that that Maker needs to be compiled at first use, and not simply being built in like `zig maker` or whatever? (I understand that it is much easier to hack on it during development this way!). As it lives in global cache and not per-project cache, it doesn't depend on any project specific options, no?

rohlem commented 2026年05月23日 15:12:12 +02:00

@bfredl I think the reasoning here would be the same as for other subcommands like zig cc, zig libc, zig objcopy etc. being compiled on-demand instead of included in the zig binary. As far as I'm aware:

• ability to optimize further (based exactly on user's target: os/environment, cpu model)
• smaller binary / download size for zig itself
• support easier hacking

@bfredl I think the reasoning here would be the same as for other subcommands like `zig cc`, `zig libc`, `zig objcopy` etc. being compiled on-demand instead of included in the `zig` binary. As far as I'm aware: - ability to optimize further (based exactly on user's target: os/environment, cpu model) - smaller binary / download size for `zig` itself - support easier hacking

alexrp commented 2026年05月23日 21:34:57 +02:00

@andrewrk wrote in #35428 (comment):

• When targeting Windows or Wine, artifact args added to Run steps used to
  modify PATH based on the set of directories containing the recursive set of
  DLL dependencies. Now this is only done for argv[0]. The motivation for the
  former behavior is unclear.

Isn't this change going to cause transitive dependency DLLs to fail to load?

@andrewrk wrote in https://codeberg.org/ziglang/zig/pulls/35428#issue-5273132: > * When targeting Windows or Wine, artifact args added to Run steps used to > modify PATH based on the set of directories containing the recursive set of > DLL dependencies. Now this is only done for argv[0]. The motivation for the > former behavior is unclear. Isn't this change going to cause transitive dependency DLLs to fail to load?

andrewrk commented 2026年05月23日 21:58:25 +02:00

Isn't this change going to cause transitive dependency DLLs to fail to load?

The difference is not in recursiveness, it is in whether the logic is done for only argv[0], or for argv[0] and also artifact args added to argv.

> Isn't this change going to cause transitive dependency DLLs to fail to load? The difference is not in recursiveness, it is in whether the logic is done for only argv[0], or for argv[0] and also artifact args added to argv.