Add option to control the behavior of the build graph builder of BLUE #268

pastor commented 2026年03月24日 10:14:59 +01:00

I've realized that we are in a good spot to make BLUE introduce a key feature that could make it a must have build system for Guix.

Guix developers often require to work in different branches. Many use worktrees to handle the different PRs, some just try to rebase on top of master, but no matter the workflow, all are affected by massive rebuilds.

Here you have a recent discussion on the topic:

• Guix devel - Local workflow.

One major pain point is that if you change branches, since master moves supper fast, you almost always touch most files leading to a massive rebuild, sometimes you are only checking a branch to later do a rebase, but that's already too late.

The point here, is that for the case of Guix, it may be desirable to configure the BLUE build graph builder to be hash based, even if it's slower than just checking timestamps, the upsides are quite good for development. It would allow us to checkout things without fear, knowing that we can always rebase on the last commit where we did the last build and everything will be properly cached.

I've realized that we are in a good spot to make BLUE introduce a key feature that could make it a must have build system for Guix. Guix developers often require to work in different branches. Many use worktrees to handle the different PRs, some just try to rebase on top of master, but no matter the workflow, all are affected by massive rebuilds. Here you have a recent discussion on the topic: - [Guix devel - Local workflow](https://lists.gnu.org/archive/html/guix-devel/2026-02/msg00176.html). One major pain point is that if you change branches, since `master` moves supper fast, you almost always touch most files leading to a massive rebuild, sometimes you are only checking a branch to later do a rebase, but that's already too late. The point here, is that for the case of Guix, it may be desirable to configure the BLUE build graph builder to be hash based, even if it's slower than just checking timestamps, the upsides are quite good for development. It would allow us to checkout things without fear, knowing that we can always rebase on the last commit where we did the last build and everything will be properly cached.

pastor commented 2026年03月24日 10:57:05 +01:00

Ludo pointed us to the guile-hashing library that implements common hashing functions, it's under a permissive license so we could embed some of the implementations if needed.

Ludo pointed us to the [guile-hashing](https://hpc.guix.info/package/guile-hashing) library that implements common hashing functions, it's under a permissive license so we could embed some of the implementations if needed.

pastor commented 2026年03月24日 11:04:35 +01:00

Computing the hash of all the files in the Guix repo has a 2.497 s overhead, it think it's worth it, specially if it's a preference that the user can set:

$ hyperfine 'cat files.txt | xargs sha256sum'
Benchmark 1: cat files.txt | xargs sha256sum
 Time (mean ± σ): 2.451 s ± 0.032 s [User: 2.330 s, System: 0.123 s]
 Range (min ... max): 2.396 s ... 2.497 s 10 runs

I'm not sure how much slower using a Guile implementation would be. With a simpler hashing function, we can get 2x performance:

$ hyperfine -i 'cat files.txt | xargs sha1sum'
Benchmark 1: cat files.txt | xargs sha1sum
 Time (mean ± σ): 1.162 s ± 0.015 s [User: 1.047 s, System: 0.119 s]
 Range (min ... max): 1.131 s ... 1.184 s 10 runs

Computing the hash of all the files in the Guix repo has a 2.497 s overhead, it think it's worth it, specially if it's a preference that the user can set: ``` $ hyperfine 'cat files.txt | xargs sha256sum' Benchmark 1: cat files.txt | xargs sha256sum Time (mean ± σ): 2.451 s ± 0.032 s [User: 2.330 s, System: 0.123 s] Range (min ... max): 2.396 s ... 2.497 s 10 runs ``` I'm not sure how much slower using a Guile implementation would be. With a simpler hashing function, we can get 2x performance: ``` $ hyperfine -i 'cat files.txt | xargs sha1sum' Benchmark 1: cat files.txt | xargs sha1sum Time (mean ± σ): 1.162 s ± 0.015 s [User: 1.047 s, System: 0.119 s] Range (min ... max): 1.131 s ... 1.184 s 10 runs ```

old commented 2026年03月24日 15:14:13 +01:00

Not sure if cryptographic hashes are necessary here. Maybe if the BLUE store gets distributed at some point, it would make sens.

Also, maybe the solution would be to cache build artifacts ourself. That way, we only hash content when timestamps have changes.

Not sure if cryptographic hashes are necessary here. Maybe if the BLUE store gets distributed at some point, it would make sens. Also, maybe the solution would be to cache build artifacts ourself. That way, we only hash content when timestamps have changes.

pastor commented 2026年03月24日 16:07:29 +01:00

@old wrote in #268 (comment):

Not sure if cryptographic hashes are necessary here. Maybe if the BLUE store gets distributed at some point, it would make sens.

The thing is that if you are in master, checkout a commit, and go back to master, usually all the timestamps change, although the contents remain the same, we want to avoid a rebuild in that case which happens quite frequently. Is there any other way to make sure that there is no need for a rebuild?

Also, maybe the solution would be to cache build artifacts ourself. That way, we only hash content when timestamps have changes.

Well, that is the case for BLUE, we will likely have a build directory.

@old wrote in https://codeberg.org/lapislazuli/blue/issues/268#issuecomment-12026127: > Not sure if cryptographic hashes are necessary here. Maybe if the BLUE store gets distributed at some point, it would make sens. The thing is that if you are in master, checkout a commit, and go back to master, usually all the timestamps change, although the contents remain the same, we want to avoid a rebuild in that case which happens quite frequently. Is there any other way to make sure that there is no need for a rebuild? > Also, maybe the solution would be to cache build artifacts ourself. That way, we only hash content when timestamps have changes. Well, that is the case for BLUE, we will likely have a `build` directory.

old commented 2026年03月24日 16:43:30 +01:00

The thing is that if you are in master, checkout a commit, and go back to master, usually all the timestamps change, although the contents remain the same, we want to avoid a rebuild in that case which happens quite frequently. Is there any other way to make sure that there is no need for a rebuild?

Usually, git checkout should set the timestamp of the change sources to the commit date not? not all files

> The thing is that if you are in master, checkout a commit, and go back to master, usually all the timestamps change, although the contents remain the same, we want to avoid a rebuild in that case which happens quite frequently. Is there any other way to make sure that there is no need for a rebuild? Usually, git checkout should set the timestamp of the change sources to the commit date not? not all files

pastor commented 2026年03月24日 16:58:57 +01:00

Yes, but the problem is that Guix master branch moves too fast. So most of the times almost all .scm files are touched, this is why for the particular case of Guix, it may be interesting to base the decision on whether a rebuild is needed depending on the content, rather than the timestamp.

Yes, but the problem is that Guix master branch moves too fast. So most of the times almost all `.scm` files are touched, this is why for the particular case of Guix, it may be interesting to base the decision on whether a rebuild is needed depending on the content, rather than the timestamp.

old commented 2026年04月01日 16:15:39 +02:00

Relevant blog posts:

https://neugierig.org/software/blog/2022/03/n2.html
https://apenwarr.ca/log/20181113

Relevant blog posts: https://neugierig.org/software/blog/2022/03/n2.html https://apenwarr.ca/log/20181113