feat(guard): mechanise self-test potency with a declared-mutant harness (#405) - #432

What & why (#405)

REQ-GUARD-GATE-EVIDENCE-002 once asserted in prose that "each was mutation-tested to show its cases are load-bearing rather than merely green." That universal was false and was withdrawn (#405): an adversarial review applied one-line mutations to two of the six guard tools and their --self-tests stayed green —

• check_verification_filters.py: filt in name → filt.split('::')[-1] in name (blind to the 16-of-60 filters carrying ::);
• check_lean_sorries.py: deleting the below-floor ratchet-notice branch (no case read what it printed).

Both holes were later closed by hand, but nothing prevented them reopening — delete either case and the suite is green again, silently. This PR ships the standing mechanism the withdrawn prose claimed to have.

The harness

tools/check_self_test_potency.py. For each mutant in a declared table:

1. read the tool source, assert the substitution applies exactly as declared (a drifted/absent/ambiguous old, or a no-op new == old, is a hard ERROR, never a silent verdict — the specific trap this repo has been bitten by);
2. assert the tool's unmodified --self-test is green (a red baseline can't be potency-judged → ERROR);
3. apply the mutation to a copy in a tempdir (the working tree is never touched), run its --self-test, and require it now fails (CAUGHT). A survivor reds the build.

Exit 0 = all caught, 1 = a survivor, 2 = a declaration/baseline fault. Stdlib-only.

Seeded with four mutants across the two tools #405 falsified, including the exact two it demonstrated (vf-colon-tail, ls-drop-below-floor). All four are CAUGHT on this tree.

Non-vacuity — executed, not asserted

• Harness self-test (12 cases): the same mutation is run against a potent stub (→ CAUGHT) and a blind stub differing only by one boundary case (→ SURVIVED). Distinct inputs, distinct outputs — a harness whose verdict ignored the target suite could not tell them apart.
• Against the real tools: removing each mutant's named catching case flips it to SURVIVED and reds the run. Confirmed for all four (vf-colon-tail, vf-none-lenient, ls-drop-below-floor, ls-drop-axiom) — so each is caught by a specific case, not an incidental SyntaxError.
• Fail-closed error handling (absent old, no-op, ambiguous count, red baseline → ERROR) is pinned by cases.

Honest ceiling

Mutation testing shows the cases present are load-bearing; it cannot show a suite is complete. A green run means "the declared mutants are all caught", never "these suites cover the hazard". The table is an allow-list of two tools / four mutants; broadening it to the other guard tools is left to a successor (noted in the REQ).

Artifacts & wiring

• REQ-GUARD-SELFTEST-POTENCY-001 (implemented, v0.40.0, traces-to REQ-GUARD-GATE-EVIDENCE-002).
• TEST-GUARD-SELFTEST-POTENCY — carries real runnable steps and satisfies/verifies links (the good -001-era convention, not the step-less regression Verification Gate is green having executed zero commands: run_verification.py counts 'skipped' as pass #403 flagged).
• Two steps in the rivet-validate job (--self-test, then the real run), beside the sibling guardrail self-tests. No new job → no required-context change.

Local verification

• check_self_test_potency.py --self-test → 12/12; real run → 4/4 CAUGHT.
• Mutant-removal experiment → each of the 4 flips to SURVIVED, harness exits 1; tree restored byte-identical.
• rivet validate → PASS; REQ/TEST appear, all cross-refs resolve (the field 'release'/'method'/'steps' INFO and missing: design-decision WARN are the repo-wide baseline, firing on every sibling REQ-GUARD-*).
• check_evidence_steps (floor 78, unchanged), check_human_scoped, check_release_plane, check_required_contexts, and every guardrail --self-test: green.
• A clean-room subagent independently re-derived the above.

🤖 Generated with Claude Code

Generated by Claude Code