Distributed build agents through Goblins #82

pastor commented 2025年09月14日 17:00:04 +02:00

In the future we could consider swapping Fibers for Goblins, which could allow distributed builds across a network of BLUE agents.

The latest post of the Spritely guys gives a nice update on their work on Shepherd: Shepherd ×ばつ Goblins update.

In the future we could consider swapping Fibers for Goblins, which could allow distributed builds across a network of BLUE agents. The latest post of the Spritely guys gives a nice update on their work on Shepherd: [Shepherd ×ばつ Goblins update](https://www.spritely.institute/news/shepherd-goblins-update.html).

pastor commented 2025年09月29日 12:42:49 +02:00

Summary of conversations from other channels:

[pastor]: I'm wondering if Guix could support partial builds somehow, and then the blue-build-system would integrate partial builds and workload distribution like they do with Guix machine offloading, but instead of offloading complete derivations it should offload build artifacts.

[pastor]: What I'm thinking is that with other build systems it's probably not feasible to do something like this because the unit of computation for Guix is the derivation. But BLUE can go down to the artifact level, so I'm wondering if we could somehow bridge this gap in the blue-build-system, so Guix can understand about what artifacts the build generates.

Given that both programs are in scheme, I'm wondering if it's something feasible. So by providing the glue code and adapting Guix for partial builds, we could get the build distributor already present in Guix to offload builds.

There were some discussions about partial Guix builds in the past, and there is also some work done in the P2P substitute network field:
https://nlnet.nl/project/Guix-P2P/

[pastor]: This is the original work of the P2P substitute network:
https://issues.guix.gnu.org/52555

[pastor]: And here is the latest news I had about partial builds:
https://lists.gnu.org/archive/html/guix-devel/2025-03/msg00236.html

Summary of conversations from other channels: [pastor]: I'm wondering if Guix could support partial builds somehow, and then the blue-build-system would integrate partial builds and workload distribution like they do with Guix machine offloading, but instead of offloading complete derivations it should offload build artifacts. [pastor]: What I'm thinking is that with other build systems it's probably not feasible to do something like this because the unit of computation for Guix is the derivation. But BLUE can go down to the artifact level, so I'm wondering if we could somehow bridge this gap in the blue-build-system, so Guix can understand about what artifacts the build generates. Given that both programs are in scheme, I'm wondering if it's something feasible. So by providing the glue code and adapting Guix for partial builds, we could get the build distributor already present in Guix to offload builds. There were some discussions about partial Guix builds in the past, and there is also some work done in the P2P substitute network field: https://nlnet.nl/project/Guix-P2P/ [pastor]: This is the original work of the P2P substitute network: https://issues.guix.gnu.org/52555 [pastor]: And here is the latest news I had about partial builds: https://lists.gnu.org/archive/html/guix-devel/2025-03/msg00236.html