WIP: Add Compilation Database Support to Zig Build System (compile_commands.json) #35462

This PR closes issue #17365 from Github.

I was wanting this functionality, and got the general idea of how to implement it from the following stale PR:

• https://github.com/ziglang/zig/pull/18391

The general idea is the same, JSON fragments are generated using Clang's -MJ[path] flag, and then merged into a single JSON file. However, I integrated this deeper into Zig's build system using a new Step.CompileCommands step type. The positive to this is that all generated JSON files are LazyPaths now, so if they aren't installed/depended upon by something, they don't get generated. The fragments themselves always get generated (when enabled) into a temporary directory since they're an artifact of the compilation itself.

New functionality:

• When creating a Step.Compile in build.zig (addExecutable(), addObject(), addLibrary(), addTest()), you now have an option .enable_compdb.
• When true, this allows you to access a generated compile_commands.json via LazyPath with a new function Step.Compile.getCompileCommandsJson().
• This is the compile_commands.json ONLY for that Step.Compile
• There is a new flag -fcompdb for zig build
• When used, this will default ALL Step.Compile to generate compile_commands.json
• For both cases, you can get a LazyPath to the merged compile_commands.json for ALL Step.Compile that had it enabled via Build.getMergedCompileCommandsJson()
• This gives an easy way to get a compile_commands.json for ALL C file compilations in a project, even across dependencies

Testing:

• I added two standalone tests at test/standalone/compdb_single and test/standalone/compdb_merged
• Check the source code for details (+ a good example of using the new API), but the general idea is I generate a bunch of compile commands files and check them against my rough expectation for how they should look

Areas for Review:

• I didn't mess with any kind of caching for the fragment files, so in theory the case where only a single C file out of many is recompiled might lead to inaccurate compile_commands.json. In practice I could never get this to happen, so I'm a little afraid enabling this feature always forces ALL C files to recompile. I don't know why that would be the case with how I've set up the step dependencies, but it's something I'd like a sanity check on.
• In the make function in CompileCommands.zig, I took the logic for writing a file to tmp that might be installed by the user later from WriteFile.zig. Specifically the code:

varrand_int:u64=undefined;io.random(@ptrCast(&rand_int));consttmp_dir_path="tmp"++Dir.path.sep_str++std.fmt.hex(rand_int);

I think this is fine, and it works in practice, but wanted to double check this is the correct way to get what I'm after.

Be aware that the build system is being reworked: #35428

I've said it before, and I will say it again: I think this is a great feature to have.

Generating compile_commands.json enables developers to:

• use the LLVM LSP clangd (useful for people using [neo]Vim, emacs, sublime, vscode)
• use static analyzers like clang-check, clang-tidy, sonarlint, pvs studio

Anecdotally, I know some people that use CMake exclusively because of this feature.
Thanks for working on this!

With the build system rework, generating compile_commands.json should not need any support in build.zig or zig itself. It would also make it possible for clangd and such be able to use zig's build graph directly as alternative to compile_commands.json

@Cloudef wrote in #35462 (comment):

With the build system rework, generating compile_commands.json should not need any support in build.zig or zig itself. It would also make it possible for clangd and such be able to use zig's build graph directly as alternative to compile_commands.json

Interesting! Do mean in the sense that it would be up to the user to consume the build graph and do it themselves? I have a rough understanding of the rework as I watched a little of Andrew's stream the other day when he was working on it. What about the rework specifically makes it so that generating/merging compile_commands fragments from compiler calls could be done outside of build.zig/Zig?

My only 2c about having it supported directly in build.zig was I really like in this implementation having compile_commands.json as a LazyDependency you can perform further processing on directly in the build system. Could open up some cool tooling possibilities!

What about the rework specifically makes it so that generating/merging compile_commands fragments from compiler calls could be done outside of build.zig/Zig?

The graph file contains all information how artifact should be produced so it should have all C/C++ file paths and flags and so on.

The graph file contains all information how artifact should be produced so it should have all C/C++ file paths and flags and so on.

That's not quite right. For example, the maker process, which consumes the configuration graph file, executes the C compiler with dep file enabled, then parses the dep file to learn the C/C++ file paths. That information is available after running the maker process.

Swapped to WIP while I take a crack at updating it to work w/ the build system re-work

Just pushed some commits that substantially refactor my approach and integrate it with the new build rework. I ditched the new CompileCommands step after further thought. JSON fragments, after all, are artifacts of a Compile step. It didn't really make sense to have the merging of these fragments as a separate step, and so that has been folded into the Maker for a Step.Compile.

This puts the current state of my feature in a half complete state. You can get merged compile_commands.json files as a LazyPath for each Step.Compile in your build.zig using compile_ptr.getCompileCommandsJson(). See the test I added in test/standalone/compdb_single for example usage.

What I'm struggling with is the feature most people want when they think of compile_commands.json, which is a merged file for all Step.Compile in your entire build graph. What I want is access to a LazyPath of this global merged file via b.getCompileCommandsJson(), where depending on this file is what triggers its creation similar to how I have it set up in Step.Compile. The shape of what I want to do is:

• User build.zig depends on global compile_commands.json via b.getCompileCommandsJson()
• For each Step.Compile in the build graph, collect the LazyPath of this step's compile_commands.json with compile_ptr.getCompileCommandsJson()
• Add to Maker merging all these files after all Step.Compile have completed

Anybody have suggestions for how to implement the "global" compile_commands.json as described above? I feel like it should live in Build.Graph but after a couple of false starts can't quite wrap my head around the how.