steve, the jobserver

steve is a GNU make-compatible system-wide jobserver implemented using FUSE. It enables running multiple build jobs simultaneously while sharing a single job pool, and therefore avoiding superlinear job count multiplication. Note that while GNU make uses a named pipe, steve uses a character device to track acquisition of job tokens and reclaim them if processes are killed without returning them.

Features

• Per-process token accounting. Steve notes how many tokens each process acquired, and reclaims them if the process exits without returning them.
• Poor man's round-robin token balancing. While steve cannot perfectly balance jobs between multiple tasks, it does attempt to split them between multiple processes running simultaneously and prevent any single program from acquiring all tokens immediately.
• Support for jobserver-level load-based scheduling with configurable minimum of jobs to sustain when load is exceeded, and per-process job limiting.
• Support for adjusting parameters at runtime via ioctls (CLI provided by stevie).

Installation

Steve uses the Meson build system. Typically, it can be built and installed using the following commands:

$ meson setup . build
$ meson install -C build

For more details, please see the Meson documentation.

Steve needs access to /dev/cuse which is normally restricted to the superuser. However, as a best security practice it is recommended to run steve using a dedicated user account. This can be achieved either by:

1. using --user {username} option when starting steve, to make it drop privileges after opening /dev/cuse, or
2. giving the user running steve permissions to open /dev/cuse.

The default device file created by CUSE (/dev/steve) is accessible only to root. The recommended way of changing permissions to these files is through udev rules. That said, world access may be undesirable, as malicious client could easily consume all the tokens and block other processes from obtaining any.

The distribution provides a number of example files that can be modified and installed:

data/steve.service

A systemd service file. It assumes that steve is installed as /usr/bin/steve and runs as steve:steve user which can access /dev/cuse.

data/steve.initd and data/steve.initd

A init.d script and conf.d file for OpenRC, with the same assumptions.

data/steve.udev

Udev rules file (needs to be renamed to *.rules) that gives the jobserver group access to /dev/steve.

data/sandbox.conf

Gentoo sandbox configuration file that gives sandboxed processes access to /dev/steve.

Usage

When started without any arguments, steve claims /dev/steve and starts serving job tokens:

$ steve
steve running on /dev/steve for 12 jobs
 at least 1 jobs will be always available

By default, steve uses the logical CPU count. The -j option can be used to override the job count:

$ steve -j6
steve running on /dev/steve for 6 jobs
 at least 1 jobs will be always available

For more options, see steve --help.

To use steve from a build system, one needs to set MAKEFLAGS appropriately. To use it for all builds in Portage, these flags can be set in make.conf:

MAKEFLAGS="--jobserver-auth=fifo:/dev/steve"

Note that to use steve with Ninja, -j must not be passed. Given that MAKEOPTS is used to provide defaults for other build systems, the best approach is to override NINJAOPTS, e.g.:

MAKEOPTS="-j12 -l13"
NINJAOPTS="-l13"

To inspect the current status, SIGUSR1 can be sent to steve:

$ killall -USR1 steve

This will cause the process to print the current token count, as well as the list of processes holding them:

steve: currently 0 tokens available out of 12
PID 221613 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 3 tokens:
 job 0x00 running for 0.0084036s
 job 0x01 running for 0.0536742s
 job 0x02 running for 0.163137s
PID 202894 (ninja -v -l16 distribution) holds 7 tokens:
 job 0x04 running for 3.0512s
 job 0x00 running for 3.05184s
 job 0x09 running for 8.47799s
 job 0x08 running for 18.8555s
 job 0x01 running for 20.4352s
 job 0x03 running for 20.7428s
 job 0x02 running for 23.3102s
PID 194264 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 0 tokens:
PID 194247 (make DESTDIR= RPATH_ENVVAR=LD_LIBRARY_PATH TARGET_SUBDIR=x86_64-pc-linux-gnu bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 datadir) holds 0 tokens:
PID 194237 (make LDFLAGS=-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu LIBPATH=/usr/lib/gcc/x86_64-pc-linux-gnu/15 STAGE1_CFLAGS=-m64 -march) holds 0 tokens:
PID 188922 (/usr/bin/python3.14 /usr/lib/python-exec/python3.14/emerge -vB --nodep llvm-core/clang sys-devel/gcc --jobs) holds 2 tokens:
 job 0x01 running for 108.73s
 job 0x00 running for 108.735s

stevie

stevie is the client for steve. It currently provides two functions:

1. Running the specified command with a job token acquired and appropriate MAKEFLAGS, e.g.:

   stevie make foo

2. Controlling steve remotely, e.g.:

   stevie --get-jobs --set-jobs 6

For the full list of getters and setters, see stevie --help.

Use in containers

steve can be used across container boundaries, provided that the container system permits sharing character devices. For systemd-nspawn, the device can be bound to the container using the following configuration:

[Files]
Bind=/dev/steve
BindReadOnly=/etc/sandbox.d/90steve

Additionally, when running the container as a service, it is necessary to permit the service to access it, e.g. via:

$ systemctl edit systemd-nspawn@${container_name}.service

Then adding:

[Service]
DeviceAllow=/dev/steve rw

Docker invocation

To run a container and make the device available inside the container, add the device mapping when invoking Docker. For example:

$ docker run ... --device /dev/steve:/dev/steve:rw <image> ...

Compatibility notes

Steve implements the POSIX jobserver protocol based on named pipes. Furthermore, due to use of CUSE it is limited to Linux. Support for other platforms is not planned at the moment.

Due to technical limitations of the Linux kernel, steve cannot use an actual named pipe; it uses a character device instead. This may cause issues with strict jobserver clients that verify that --jobserver-auth specifies an actual named pipe. However, upstreams can generally be convinced to allow character devices as well.

Per-process accounting implies that steve requires that the same process acquired and returns job tokens. If a process "passes" a job token to a different process and exits, steve reclaims the token immediately. Attempting to return a token not acquired by the process returning it results in a ENOSPC error being returned to the client.

To work around bugs in existing software, steve permits a single job token to be "returned" by a new client prior to acquiring any clients. However, this behavior is not guaranteed to be preserved in the future, and new clients must not rely on it.