...EQ-316)
Two CI signals were sending readers to the wrong conclusion, and both had
already produced a published misattribution.
REQ-317 — the liveness probe had three defects, all fixed:
 1. Its diagnose hint told every reader to run
 `gh api repos/<repo>/actions/runners --jq .total_count` and read `0 = pool
 offline`. Runners here are ORG-registered, so that command returns 0 on a
 perfectly healthy pool and can never say otherwise. The alert's own hint
 was the trap.
 2. It reported a duration and nothing else, so hosted starvation and a dead
 self-hosted pool produced identical alerts — and hosted capacity appears
 in the runners API at NO scope, so no runner count can tell them apart.
 3. It counted queue age on runs that had already COMPLETED: #867 reported
 "queued 177m" for a cancelled run that was never going to start.
REQ-316 — a job killed by a fleet restart reports `failure` with NO failed
step, because the steps end `skipped`. GitHub renders that identically to a
broken test, which is how a wrong cause got published on #855.
Both are now shell functions in tools/ci-diagnose/diagnose.sh with fixtures:
 classify_stall no-queue | hosted-starved | pool-offline | label-saturated
 | capacity-available
 classify_failure job-passed | runner-vanished | setup-failed | job-failed
A FIFTH failure mode turned up while building this and is covered: label-
partitioned saturation. On 2026年09月02日 the pool read online=12 busy=8 — a third
of the fleet idle — while every rust-cpu job queued, because the four idle
runners carried only lean-mem. A global busy-versus-online count cannot see
that, so capability is computed PER LABEL against the labels of what is queued.
Validated against live data, not only fixtures: with 5 hosted and 3 rust-cpu
jobs queued and only lean-mem and light runners idle, classify_stall returns
label-saturated; the control that filters to just the hosted jobs returns
hosted-starved. classify_failure on the real Kani job from #888 returns
setup-failed — the same discriminator applied by hand all cycle, now mechanical.
Negative-controlled: ignoring per-label capability, losing the runner-vanished
case, and dropping the hosted short-circuit each redden the oracle.
The tests run in the YAML Lint job, which IS in CI Gate's needs. A classifier
that never executes is documentation, not a gate. They are shell, and their
`rivet: verifies` markers are picked up by `coverage --tests` only because
REQ-319 taught the scanner to read shell scripts earlier in this same round —
before that they would have been invisible.
REQ-316 is SPLIT, decided with the maintainer. This carries the legibility
half. Finding the cause on the runner hosts — unattended upgrades, a systemd
timer, config management or an autoscaler reclaim, all consistent with the
04:30-05:00 UTC window both incidents fall in — needs shell access to those
machines and is deferred to REQ-323 at v0.37.0. Recorded, not dropped.
Confirmed with the diagnose oracle (10 cases), cargo fmt --check, cargo test
--workspace (exit 0, 66 ok), rivet validate, rivet docs check, yamllint on the
CI paths — all exit 0. Both workflow files parse as YAML and every embedded
shell block passes `bash -n`.
Implements: REQ-316, REQ-317
Verifies: REQ-316, REQ-317

...EQ-316)
Two CI signals were sending readers to the wrong conclusion, and both had
already produced a published misattribution.
REQ-317 — the liveness probe had three defects, all fixed:
 1. Its diagnose hint told every reader to run
 `gh api repos/<repo>/actions/runners --jq .total_count` and read `0 = pool
 offline`. Runners here are ORG-registered, so that command returns 0 on a
 perfectly healthy pool and can never say otherwise. The alert's own hint
 was the trap.
 2. It reported a duration and nothing else, so hosted starvation and a dead
 self-hosted pool produced identical alerts — and hosted capacity appears
 in the runners API at NO scope, so no runner count can tell them apart.
 3. It counted queue age on runs that had already COMPLETED: #867 reported
 "queued 177m" for a cancelled run that was never going to start.
REQ-316 — a job killed by a fleet restart reports `failure` with NO failed
step, because the steps end `skipped`. GitHub renders that identically to a
broken test, which is how a wrong cause got published on #855.
Both are now shell functions in tools/ci-diagnose/diagnose.sh with fixtures:
 classify_stall no-queue | hosted-starved | pool-offline | label-saturated
 | capacity-available
 classify_failure job-passed | runner-vanished | setup-failed | job-failed
A FIFTH failure mode turned up while building this and is covered: label-
partitioned saturation. On 2026年09月02日 the pool read online=12 busy=8 — a third
of the fleet idle — while every rust-cpu job queued, because the four idle
runners carried only lean-mem. A global busy-versus-online count cannot see
that, so capability is computed PER LABEL against the labels of what is queued.
Validated against live data, not only fixtures: with 5 hosted and 3 rust-cpu
jobs queued and only lean-mem and light runners idle, classify_stall returns
label-saturated; the control that filters to just the hosted jobs returns
hosted-starved. classify_failure on the real Kani job from #888 returns
setup-failed — the same discriminator applied by hand all cycle, now mechanical.
Negative-controlled: ignoring per-label capability, losing the runner-vanished
case, and dropping the hosted short-circuit each redden the oracle.
The tests run in the YAML Lint job, which IS in CI Gate's needs. A classifier
that never executes is documentation, not a gate. They are shell, and their
`rivet: verifies` markers are picked up by `coverage --tests` only because
REQ-319 taught the scanner to read shell scripts earlier in this same round —
before that they would have been invisible.
REQ-316 is SPLIT, decided with the maintainer. This carries the legibility
half. Finding the cause on the runner hosts — unattended upgrades, a systemd
timer, config management or an autoscaler reclaim, all consistent with the
04:30-05:00 UTC window both incidents fall in — needs shell access to those
machines and is deferred to REQ-323 at v0.37.0. Recorded, not dropped.
Confirmed with the diagnose oracle (10 cases), cargo fmt --check, cargo test
--workspace (exit 0, 66 ok), rivet validate, rivet docs check, yamllint on the
CI paths — all exit 0. Both workflow files parse as YAML and every embedded
shell block passes `bash -n`.
Implements: REQ-316, REQ-317
Verifies: REQ-316, REQ-317

...EQ-316) (#890)
Two CI signals were sending readers to the wrong conclusion, and both had
already produced a published misattribution.
REQ-317 — three defects in the liveness probe, all fixed. Its diagnose hint told
every reader to run a REPO-scope actions/runners total_count query and read
0 = pool offline; runners here are ORG-registered, so that returns 0 on a
healthy pool and can never say otherwise — the alert's own hint was the trap. It
reported a duration and nothing else, so hosted starvation and a dead
self-hosted pool produced identical alerts, and hosted capacity appears in the
runners API at NO scope. And it counted queue age on runs that had already
completed: #867 reported "queued 177m" for a cancelled run.
REQ-316 — a job killed by a fleet restart reports failure with NO failed step,
because the steps end skipped. GitHub renders that identically to a broken test,
which is how a wrong cause got published on #855.
Both are now shell classifiers with fixtures. A fifth failure mode turned up
while building this and is covered: label-partitioned saturation, measured
2026年09月02日 as online=12 busy=8 while every rust-cpu job queued, because the four
idle runners carried only lean-mem. Capability is computed PER LABEL against the
labels of what is queued.
Validated against live data, not only fixtures: 5 hosted and 3 rust-cpu jobs
queued with only lean-mem and light idle returns label-saturated, the control
filtering to just the hosted jobs returns hosted-starved, and the real Kani job
from #888 returns setup-failed. That last classification was then used to
re-derive #839's failure rate: all 26 Kani failures across 60 jobs are the
install action, none is a proof break, and none falls in the needrestart window.
The tests run in the YAML Lint job, which IS in CI Gate's needs — a classifier
that never executes is documentation, not a gate. Their markers are read by
coverage --tests only because REQ-319 taught the scanner to read shell scripts
earlier in this same round.
REQ-316 is SPLIT with the maintainer: this carries the legibility half. The
root cause was found on #862 (needrestart restarting github-runner@ under
unattended-upgrades) and is recorded on REQ-323 as implemented-not-verified,
since suppression of a real restart has not been demonstrated yet.
Implements: REQ-316, REQ-317
Verifies: REQ-316, REQ-317
Refs: REQ-323, REQ-319