build server protocol #35538

andrewrk commented 2026年05月30日 01:24:15 +02:00

Followup from #35428.

There is already https://github.com/ziglang/zig/issues/615 to track serving Compilation information. This is for stuff like type information, looking up definitions, refactoring, etc.

By contrast, this issue is to implement the build system protocol which is a higher layer, exposed by zig build --listen=-, which provides a machine interface to the build system. Thanks to #35428 the entire build graph is available in read-only serialized form, however, what's lacking is interactivity. Connected clients, which are IDEs and related tooling, need access to generated file paths, error bundles and other messages, and success status for each step. Furthermore, they need to be able to control the maker process by issuing a rebuild command.

The CLI flag will be --listen=- which turns stdin/stdout into the build system protocol. --listen=[ip] will be reserved for listening via network socket, but not implemented at first. stderr reserved for troubleshooting the maker process itself. This will cause the maker process to be long-running, the same as if --webui, --watch, or --fuzz is passed. However, the process should not panic or otherwise crash if file watching cannot be performed. Instead the file watching failure should be communicated via the protocol to the parent process. File watching is still an independent feature that can be enabled or disabled simultaneously with the build system protocol.

Unlike some of the other protocols only used for zig-version-locked processes talking to each other, this protocol faces third party software. As such, it needs to be independently versioned and benefits from ABI stability. For now, the plan is to have an integer protocol version and bump it whenever any backwards incompatible changes to the protocol are made. The configuration file format also should have this version.

MVP Protocol Specifics

This is the issue close criteria, which should unblock ZLS from updating to latest Zig.

Upon first connection, server sends to client:

• protocol version
• path to configuration file

Server does not kick off a build until client sends a build message. If any -D CLI flags or step names are provided along with --listen, maker process reports an error and exits.

Client to server messages:

• build (protocol equivalent of zig build ...)
  • set of step names
  • set of options (equivalent to -D on the command line)

Server to client messages:

• build started (might trigger due to explicit rebuild, or due to file system watching)
• build complete
  • for each step:
    • status
    • error bundle / other error information
    • peak rss and duration
  • set of generated files that changed
• configuration file changed, path to new configuration file. All messages after this reference the new configuration.
• log messages (info, warning, debug, error) pertaining to the maker process itself. For example, error, file watching could not be initialized.

When closing this issue, file new issues as follow-up enhancements:

Followup Enhancement: Run Proxy

IDE needs to be able to drive a particular child process execution so that it can let the user drive a debugger (think breakpoints on source locations).

• client to server messages:
  • enable/disable run proxy for a particular run step
  • ran command
• server to client messages:
  • run command request
    • argv, environ
    • need to send a ran command response

Followup Enhancement: Unit Testing

• provide (live!) source coverage information
• ability to choose which unit tests to run
• fuzz testing status updates & other integration

Followup from #35428. There is already https://github.com/ziglang/zig/issues/615 to track serving Compilation information. This is for stuff like type information, looking up definitions, refactoring, etc. By contrast, this issue is to implement the **build system protocol** which is a higher layer, exposed by `zig build --listen=-`, which provides a machine interface to the build system. Thanks to #35428 the entire build graph is available in read-only serialized form, however, what's lacking is interactivity. Connected clients, which are IDEs and related tooling, need access to generated file paths, error bundles and other messages, and success status for each step. Furthermore, they need to be able to control the maker process by issuing a rebuild command. The CLI flag will be `--listen=-` which turns stdin/stdout into the build system protocol. `--listen=[ip]` will be reserved for listening via network socket, but not implemented at first. stderr reserved for troubleshooting the maker process itself. This will cause the maker process to be long-running, the same as if --webui, --watch, or --fuzz is passed. However, the process should not panic or otherwise crash if file watching cannot be performed. Instead the file watching failure should be communicated via the protocol to the parent process. File watching is still an independent feature that can be enabled or disabled simultaneously with the build system protocol. Unlike some of the other protocols only used for zig-version-locked processes talking to each other, this protocol faces third party software. As such, it needs to be independently versioned and benefits from ABI stability. For now, the plan is to have an integer protocol version and bump it whenever any backwards incompatible changes to the protocol are made. The configuration file format also should have this version. ## MVP Protocol Specifics This is the issue close criteria, which should unblock ZLS from updating to latest Zig. Upon first connection, server sends to client: - protocol version - path to configuration file Server does *not* kick off a build until client sends a build message. If any `-D` CLI flags or step names are provided along with `--listen`, maker process reports an error and exits. Client to server messages: - build (protocol equivalent of `zig build ...`) - set of step names - set of options (equivalent to `-D` on the command line) Server to client messages: - build started (might trigger due to explicit rebuild, or due to file system watching) - build complete - for each step: - status - error bundle / other error information - peak rss and duration - set of generated files that changed - configuration file changed, path to new configuration file. All messages after this reference the new configuration. - log messages (info, warning, debug, error) pertaining to the maker process itself. For example, error, file watching could not be initialized. When closing this issue, file new issues as follow-up enhancements: ## Followup Enhancement: Run Proxy IDE needs to be able to drive a particular child process execution so that it can let the user drive a debugger (think breakpoints on source locations). * client to server messages: - enable/disable run proxy for a particular run step - ran command * server to client messages: - run command request - argv, environ - need to send a ran command response ## Followup Enhancement: Unit Testing - provide (live!) source coverage information - ability to choose which unit tests to run - fuzz testing status updates & other integration

pentuppup commented 2026年05月30日 02:05:16 +02:00

Upon first connection, server sends to client:

• protocol version
• path to configuration file

would it maybe make more sense for the server to send the serialized configuration over the wire instead of needing to go through the file? that would simplify the process a bit since the client only ever needs to interact with the server itself and theres no possibility of TOCTOU

> Upon first connection, server sends to client: > * protocol version > * path to configuration file would it maybe make more sense for the server to send the serialized configuration over the wire instead of needing to go through the file? that would simplify the process a bit since the client only ever needs to interact with the server itself and theres no possibility of TOCTOU

andrewrk commented 2026年05月30日 02:09:49 +02:00

Hmm yes I think that makes sense.

Hmm yes I think that makes sense.

Techatrix commented 2026年05月30日 02:33:02 +02:00

Is it intentional to call this issue "build server protocol" rather than "build system protocol"? The issue description mentions the latter.

Is it intentional to call this issue "build server protocol" rather than "build system protocol"? The issue description mentions the latter.

anthon commented 2026年05月30日 08:25:36 +02:00

@andrewrk wrote in #35538 (comment):

build (protocol equivalent of zig build ...)

* set of step names
* set of options (equivalent to `-D` on the command line)

Can the valid set of steps names, of (system integration) options, be queried from the server? Or does the client need to extract that from build.zig and the server assumes only valid ones are passed to it?

For commandline completion ( in zigclc) I need to be able to retrieve build.zig dependend elements (steps, options, and in addition to that the system integrations). I currently parse that information from the output of zig build -h, but that is somewhat fragile. I will need to continue to do that for current and older Zig releases (which exact zig to invoke is based on determined based on zigclc parsing build.zig.zon), but I would be interested to get that information, at least for future zig versions, in a more direct way if possible.

@andrewrk wrote in https://codeberg.org/ziglang/zig/issues/35538#issue-5455190: > build (protocol equivalent of `zig build ...`) > > * set of step names > > * set of options (equivalent to `-D` on the command line) Can the valid set of steps names, of (system integration) options, be queried from the server? Or does the client need to extract that from `build.zig` and the server assumes only valid ones are passed to it? For commandline completion ( in [zigclc](https://codeberg.org/anthon/zigclc)) I need to be able to retrieve `build.zig` dependend elements (steps, options, and in addition to that the system integrations). I currently parse that information from the output of `zig build -h`, but that is somewhat fragile. I will need to continue to do that for current and older Zig releases (which exact zig to invoke is based on determined based on zigclc parsing `build.zig.zon`), but I would be interested to get that information, at least for future zig versions, in a more direct way if possible.

1off commented 2026年05月30日 09:35:28 +02:00

@anthon

Your stated use-case doesn't require a server.
With bfc1484c0f merged:

zig build --print-configuration-path

How to load the file and get the top_level_steps
Iterating over the steps, available and system integration options

@anthon Your stated use-case doesn't require a server. With https://codeberg.org/ziglang/zig/commit/bfc1484c0fe3adc3b45791f6ae18a362980c991e merged: ``` zig build --print-configuration-path ``` [How to load the file and get the top_level_steps](https://codeberg.org/ziglang/zig/src/commit/09ad5c029a24f77e3de3586fa10164374c4c9ac5/lib/compiler/Maker.zig#L473-L496) [Iterating over the steps, available and system integration options](https://codeberg.org/ziglang/zig/src/commit/09ad5c029a24f77e3de3586fa10164374c4c9ac5/lib/compiler/Maker/ScannedConfig.zig#L188-L282)

matklad commented 2026年05月30日 14:12:35 +02:00

enable/disable run proxy for a particular run step

Humans also need this, not only IDEs. As far as I understand, today there's no straightforward way to debug addRunArtifact directly, you need to edit build.zig to either add a separate top-level step to compile/install artifact (but then you loose addRunArtifact runtime configuration, like env or cli args), or something like addSystemCommand(&.{"lldb", "--", artifact.

>enable/disable run proxy for a particular run step Humans also need this, not only IDEs. As far as I understand, today there's no straightforward way to debug `addRunArtifact` directly, you need to edit build.zig to either add a separate top-level step to compile/install artifact (but then you loose addRunArtifact runtime configuration, like env or cli args), or something like `addSystemCommand(&.{"lldb", "--", artifact`.

1off commented 2026年06月02日 14:10:55 +02:00

Apologies for being a scatterbrain, but any time I'm wrong -- do let me know.

My understanding:
A. Configuration is one and done, it may be cached or it will be generated on zig build --listen=-
B. Cannot provide step names or option(s) on the cmd line along with --listen

As someone who's done support for 3rd party tools, a use-case comes to mind.
This can be simply demonstrated with a zig init project and a few added lines:

// Insert after the `const exe = ...` declconstexe_options=b.addOptions();constexe_options_mod=exe_options.createModule();exe.root_module.addImport("build_options",exe_options_mod);constxtra_opts=b.option(bool,"xtra-opts","Add some extra options")orelsefalse;if(xtra_opts){constxtra_optA=b.option(bool,"xtra-optA","Fancy optA")orelsetrue;constxtra_optB=b.option(bool,"xtra-optB","How many can there be?")orelsefalse;exe_options.addOption(bool,"xtra-optA",xtra_optA);exe_options.addOption(bool,"xtra-optB",xtra_optB);}

> zig build --help | grep Dxtra
 -Dxtra-opts=[bool] Add some extra options
> zig build --help -Dxtra-opts | grep Dxtra
 -Dxtra-opts=[bool] Add some extra options
 -Dxtra-optA=[bool] Fancy optA
 -Dxtra-optB=[bool] How many can there be?
> zig build --print-configuration-path
.zig-cache/c/ffb799a733569617b6b2c55b7cee17cc
> zig build -Dxtra-opts --print-configuration-path
.zig-cache/c/8c5b473f074b28cd48fdc9c5b01fc339

From the perspective of a 3rd party tool they'd always have to do two invocations whenever an option changes
(--print-configuration-path in order to exit early, without building):

zig build --print-configuration-path -Dopt1 -Dopt2 ...
zig build --listen // provide the step name(s) and opt(s), again, over the protocol

Thoughts?

Apologies for being a scatterbrain, but any time I'm wrong -- do let me know. My understanding: A. Configuration is one and done, it may be cached or it will be generated on `zig build --listen=-` B. Cannot provide step names or option(s) on the cmd line along with `--listen` As someone who's done support for 3rd party tools, [a use-case](https://github.com/ziglang/vscode-zig/issues/472) comes to mind. This can be simply demonstrated with a `zig init` project and a few added lines: ```zig // Insert after the `const exe = ...` decl const exe_options = b.addOptions(); const exe_options_mod = exe_options.createModule(); exe.root_module.addImport("build_options", exe_options_mod); const xtra_opts = b.option(bool, "xtra-opts", "Add some extra options") orelse false; if (xtra_opts) { const xtra_optA = b.option(bool, "xtra-optA", "Fancy optA") orelse true; const xtra_optB = b.option(bool, "xtra-optB", "How many can there be?") orelse false; exe_options.addOption(bool, "xtra-optA", xtra_optA); exe_options.addOption(bool, "xtra-optB", xtra_optB); } ``` ``` > zig build --help | grep Dxtra -Dxtra-opts=[bool] Add some extra options > zig build --help -Dxtra-opts | grep Dxtra -Dxtra-opts=[bool] Add some extra options -Dxtra-optA=[bool] Fancy optA -Dxtra-optB=[bool] How many can there be? > zig build --print-configuration-path .zig-cache/c/ffb799a733569617b6b2c55b7cee17cc > zig build -Dxtra-opts --print-configuration-path .zig-cache/c/8c5b473f074b28cd48fdc9c5b01fc339 ``` From the perspective of a 3rd party tool they'd always have to do two invocations whenever an option changes (`--print-configuration-path` in order to exit early, without building): ``` zig build --print-configuration-path -Dopt1 -Dopt2 ... zig build --listen // provide the step name(s) and opt(s), again, over the protocol ``` Thoughts?

andrewrk commented 2026年06月02日 21:52:25 +02:00

(assigning to myself to indicate that I intend to help get this landed by the 0.17.0 tag)

(assigning to myself to indicate that I intend to help get this landed by the 0.17.0 tag)

CalebLarsen commented 2026年07月01日 20:32:08 +02:00

Heads up, there is a naming conflict for "Build Server Protocol" with this LSP-inspired project.

My understanding of the other project is that enough of it assumes a Java-like structure that it hasn’t picked up adoption outside of the Scala community. In my opinion, it would not be worthwhile to use their protocol, but be aware that Zig’s build server protocol should probably not literally be called BSP.

Heads up, there is a naming conflict for "Build Server Protocol" with [this LSP-inspired project](https://build-server-protocol.github.io/). My understanding of the other project is that enough of it assumes a Java-like structure that it hasn’t picked up adoption outside of the Scala community. In my opinion, it would not be worthwhile to use their protocol, but be aware that Zig’s build server protocol should probably not literally be called BSP.

andrewrk commented 2026年07月01日 21:26:42 +02:00

Thanks for sharing.

This isn't so much a naming conflict as it is the same words used to describe the same concept. It would be like saying that calling Zig "a programming language" has a naming conflict with APL. "zig build server protocol" is not a proper noun, it's just a literal description of the thing.

Thanks for sharing. This isn't so much a naming conflict as it is the same words used to describe the same concept. It would be like saying that calling Zig "a programming language" has a naming conflict with [APL](https://en.wikipedia.org/wiki/APL_(programming_language)). "zig build server protocol" is not a proper noun, it's just a literal description of the thing.

mrakh commented 2026年07月02日 21:57:49 +02:00

I have one request, since it's not explicitly mentioned in the issue: if the --listen flag will have support for network sockets, then it should also support Unix sockets (including abstract sockets in Linux).

I have one request, since it's not explicitly mentioned in the issue: if the `--listen` flag will have support for network sockets, then it should also support Unix sockets (including abstract sockets in Linux).