ci: per-PR Mutation Testing occupies the scarce lean-mem pool for up to 4h with a ~4% success rate #400

@avrabe

Summary

The mutants (Mutation Testing) job in ci.yml runs on every code-touching PR/push, on the scarce 4-runner lean-mem pool, with timeout-minutes: 240. By the workflow's own measurements it succeeds ~1 run in 25. The result is that a 4-runner pool shared by the whole org is occupied for hours by a job that is usually cancelled before it finishes — starving the gating jobs (Miri, Verification Gate, Coverage) that queue behind it.

Credit where due: the exhaustive full-workspace run was already correctly split out to mutants-weekly.yml. This issue is only about the per-PR gating job that remained on lean-mem.

Evidence

From the workflow's own comment (ci.yml, mutants job):

"The only run of this job to reach success in the last 25 CI runs took 182.3 minutes; the other 16 were cancelled by a later push."

So ~1 success in 25 runs, at ~3 hours each, with 16 cancelled mid-flight — each of those still held a lean-mem runner until the cancellation landed. That's mostly wasted occupancy of the org's scarcest CI resource.

From a 14-day org-wide runner audit (sampled Actions jobs, grouped by runner class):

runner class	jobs	median wait	p95 wait	>5 min	non-success
lean-mem (4 runners)	284	64 min	~23 h	56%	42%
rust-cpu (7 runners)	1129	0.3 min	52 min	31%	14%
light (1)	489	0.1 min	4.6 min	4%	8%

lean-mem is the fleet's only contended class — rust-cpu was ~86% idle in the same window. spar's share of the lean-mem load in that sample:

repo · job	instances	trigger	med wait	fail%
spar · Miri	47	PR/push	7 min	21%
spar · Mutation Testing	46	PR/push	12 min	28%

Live observation (2026年08月07日 ~00:30Z): spar: Mutation Testing was occupying 2 of the 4 lean-mem runners simultaneously.

Why it matters

lean-mem is 4 runners for the entire org (rivet Miri/Verus, witness, sigil fuzz, spar Miri/mutants all target it). A per-PR job with a 4-hour ceiling and a ~4% success rate is the single easiest thing to move off it. Meanwhile rust-cpu has 7 runners and sat ~86% idle over the audit window — there is spare capacity right next door.

Suggested fixes (cheapest first)

1. Move the gating mutants job to rust-cpu. It's already narrowed to spar-analysis (not the full workspace), so the RAM argument for lean-mem is much weaker than it was. One-line runs-on change; frees the scarce pool immediately.
2. Cut timeout-minutes: 240. Four hours on a shared 4-runner pool is a long time to hold a slot for a job that is usually superseded. Something closer to the measured 182 min — or lower with (3) — bounds the damage.
3. Scope to changed code — cargo mutants --in-diff so PR-time mutation shrinks with the diff instead of re-running the whole crate. This likely also fixes the cancellation rate, since faster runs finish before the next push.
4. Optional: keep it on lean-mem but make it non-blocking/nightly, leaving mutants-weekly.yml as the thorough pass.

(1) alone resolves the contention; (1)+(3) would also make the job actually useful per-PR.

Minor: stale comment

Both the miri and mutants job comments describe lean-mem as "24 G MemoryHigh". It is currently MemoryHigh=32G, MemoryMax=48G (a hard cap was added 2026年06月25日 after a runaway test OOM-killed the host). Worth updating so the sizing rationale reflects reality.

Acceptance

• Per-PR mutation testing no longer occupies lean-mem, or is bounded so it can't hold a slot for hours.
• lean-mem median job wait drops back to minutes, so Miri / Verification Gate / Coverage stop queueing behind it.

Related: pulseengine/rivet#523 (same pattern in rivet — per-PR mutants on lean-mem), pulseengine/sigil#146 (nightly fuzz fan-out on lean-mem), rivet#509 (self-hosted pool fragility).

---

Filed from a 14-day runner-utilization audit of the pulseengine self-hosted fleet plus live observation. Suggested labels: ci, infrastructure.