strict:true with no merge queue makes every concurrent PR's heavy CI dead work #380

@avrabe

The mechanism

main enforces required_status_checks.strict = true ("require branches to be up to date before merging") and there is no merge queue (gh api repos/pulseengine/spar/rulesets returns empty).

Consequence: only the first PR to merge can use the CI results it already has. The moment it lands, every other open PR is BEHIND. Updating a branch creates a new commit, a new commit fires synchronize, and synchronize re-runs all 18 required contexts. So for N concurrent PRs, N−1 full CI cycles are discarded — every round.

This is not hypothetical; it is the current state of the repo.

Measured cost (2026年07月30日)

Runner topology — 12 runner processes on one host, org-shared:

label	runners
lean-mem	1, 2, 3, 4
rust-cpu	5, 6, 7, 9, 10, 11, 12
light	8
netns	12
podman	9 only

Mutation Testing requires lean-mem, so it draws on a 4-runner pool, and it has no timeout-minutes (GitHub's 360-minute default applies). Successful durations over the last 40 CI runs: n=15, min 35.2 min, median 68.9 min, max 118.4 min.

With 7 PRs open, 4 concurrent cargo mutants -p spar-analysis --jobs 4 runs were executing on all 4 lean-mem runners at once — on a single physical host, which is precisely the contention the job's own comment warns about (# lean-mem — many parallel cargo invocations, RAM pressure under -j 4). Under that contention the run on main reached 170 minutes, 44% past the historical maximum.

Six of those seven runs are dead work by construction.

Cancelling doomed runs does not help: the freed runner is claimed by the next queued — equally doomed — mutation job within seconds. Verified by doing it.

Secondary finding: the podman runner is not reserved

Trace-Topology Fixture Generation needs runs-on: [self-hosted, linux, x64, podman], and exactly one runner carries podman (runner-9). But runner-9 is also in the general rust-cpu pool, so any ordinary Clippy/Test/Proptest job can occupy the only host the fixture workflow can ever use. A workflow_dispatch of it sat queued 25+ minutes behind unrelated Rust work. Not a deadlock — the label set does exist — but the scarce uniquely-labelled runner is unprotected.

Options

1. Enable a GitHub merge queue on main. This is the mechanism designed for exactly this: PRs are tested speculatively against the projected merge result, batched, and merged when green — so N PRs cost ~1 CI cycle instead of N. Requires adding a merge_group trigger to the workflows.
2. Drop strict and rely on the checks plus Verification Gate to catch semantic conflicts. Cheaper, but reintroduces the "both PRs green, merged result broken" risk that strict exists to prevent.
3. Keep strict, but stop paying for it: don't run the heavy tier on a PR until it is next in line to merge (e.g. gate the heavy jobs behind a label such as ready-to-merge). Keeps the guarantee, removes the dead work.
4. Give podman its own runner (or remove rust-cpu from runner-9) so the fixture workflow cannot be starved by ordinary Rust jobs.

Option 1 addresses the root cause; option 4 is independent and cheap.

Not doing anything unilaterally

Each of these changes branch-protection or runner policy, which is a maintainer decision — filing with the measurements rather than picking one.

Related: #379 (path filter misses rivet.yaml, so pure-manifest PRs pay the full heavy tier — compounds this).