Research-harden Gauntlet, Meditate, Council, and Gems before empirical benches #25

@Kitahl

Purpose

Freeze the next implementation/research target before empirical benchmark work. The names remain unchanged; the goal is to make each named component correspond to a mechanically distinct, evidence-grounded method rather than merely a role prompt.

This issue is pre-benchmark. It records implementation defects, research questions, candidate mathematical formalisms, and open-source tools to evaluate. It does not claim that any proposed change improves outcomes until tested prospectively.

1. Process Assurance Framework / Gauntlet

Current implementation gaps

• skills/infinity-gauntlet/SKILL.md specifies ten operations, but tools/gauntlet_boundary.py automatically detects only frame and costume; the other eight operations are primarily specification-level.
• costume depends on the optional LLM judge. When no judge model/API is available, lexical detection can nominate costume but the judge returns false, producing a silent no-op rather than UNKNOWN / UNAVAILABLE / a conservative warning.
• Boundary history is only a short lossy similarity window plus a bounded tool-call tail. That is useful for privacy but too weak to represent the full process state named by the ten-operation specification.
• Fixed lifetime fire budgets (frame, costume) can suppress later independent incidents; use event identity/refractory logic instead of global exhaustion.
• tools/gauntlet_monitor.py monitors Git HEAD plus configured governing-file hashes, but not tool/dependency versions, remote authorities, source timestamps, experiment state, or claim/evidence status.
• tools/verify_ledger.py verifies that evidence paths exist; it does not verify evidence content hashes, provenance, freshness, independence, claim–evidence entailment, or verifier identity. This is an existence check, not yet a strong assurance case.

Research direction

Treat Gauntlet as a runtime assurance / runtime verification layer over a research process:

• typed event stream and explicit process-state model;
• monitorability/observability metadata for every operation;
• PASS | FAIL | UNKNOWN | UNAVAILABLE semantics rather than silent non-firing when evidence is missing;
• operation-specific monitors derived from named process hazards;
• content-addressed claim/evidence graph (claim → argument → evidence → verifier → provenance/hash);
• mutation/hazard injection tests for every monitor;
• cost-aware meta-controller for expensive checks.

Research anchors:

• Runtime verification surveys and monitorability/uncertain-trace work.
• Goodloe & Havelund, High-Integrity Runtime Verification (2024).
• Gautham et al., STPA-Driven Multilevel Runtime Monitoring for In-Time Hazard Detection (2022).
• Rushby, Runtime Certification (2008): explicit goals, evidence, arguments + runtime verification.
• Taleb, Hallé & Khoury, Uncertainty in runtime verification: A survey (2023).

Open-source/tool leads to inspect:

• R2U2/r2u2 — stream/runtime monitoring with temporal specifications.
• nickovic/rtamt — online/offline temporal-logic monitoring.
• MedSecurance/Assurance — Open Evidential Tool Bus for claims/evidence/assurance cases.
• david-a-wheeler/verocase — text-first assurance cases with GSN/SACM generation.

2. Meditate / Decision Preflight Protocol

Current implementation gaps

skills/meditate/SKILL.md defines a useful five-stage protocol (STILL → GROUND → ORIENT → WEIGH → RELEASE) but there is no dedicated runtime/state machine, no trigger detector, no decision-state artifact, no quantitative stopping/next-computation rule, and no regression suite for the protocol.

RELEASE currently says to choose the single action with the highest information/progress value, but does not define that quantity. This is the main opportunity for a mathematically grounded implementation.

Research direction

Model Meditate as resource-rational metareasoning / value-of-computation rather than generic reflection.

Candidate normative rule:

VOC(a) = E[max_d EU(d | O_a)] - max_d EU(d) - C(a)

where a is a computation/information-gathering action, O_a its possible observation, d the downstream decision, and C(a) its time/tool/cognitive cost. Execute another reasoning/information step only when its estimated value is positive; otherwise release the best current action.

When calibrated probabilities/utilities are unavailable, use an explicitly ordinal/proxy scoring model and label it as heuristic rather than fake precision.

Suggested DecisionState artifact:

• goal + success condition;
• authoritative artifacts with hashes/versions;
• supported facts + evidence;
• assumptions + refuters/cost-if-wrong;
• unknowns + decision sensitivity;
• candidate actions + cost, reversibility, expected information/progress/risk reduction;
• current blocker;
• release/continue decision and rationale.

Potential triggers should be conditional, not ceremonial: high irreversibility/stakes, stale authority, repeated failure/fixation, decision-sensitive unknowns, or major disagreement among candidate actions.

Research anchors:

• Russell & Wefald, Principles of Metareasoning (1991): expected utility/value of computation for selecting computational actions.
• Lieder & Griffiths, resource-rational analysis (2019/2020).
• Value-of-Information literature (e.g. Jackson et al., Annual Review of Statistics and Its Application 2022).
• Decision aids/guided reflection/cognitive forcing systematic reviews.
• Fixation/defixation systematic review (Wang, Okada & Takagi 2023): defixation strategy depends on fixation source and problem type.
• Harmon & Walden (Information Systems Research, 2025/2026): a delay can improve decisions when task-relevant information is present during the pause, but can hurt when it is not — useful boundary condition for STILL + GROUND.

3. Council / Evidence Review Panel

Current implementation gaps

The Council specification is stronger than the closest executable panel helper:

• the skill specifies 3–6 artifact-derived roles, distinct questions/methods/evidence, a skeptic, commit-reveal, disjoint evidence where feasible, and a matched direct control;
• tools/fsa_bots.py runs two models with the same generic red-team system prompt, the same candidate brief, cross-critique, and synthesis by model A;
• it does not enforce artifact-derived role diversity, disjoint evidence acquisition, a same-budget direct control, confidence calibration, evidence-correlation measurement, or independent commit artifacts;
• tools/snap.py provides multiple proposals/critiques but is not an implementation of Council’s matched-control/evidence-independence contract.

Research direction

Implement a dedicated tools/council.py whose unit is not "agent opinion" but (claim, evidence, method, confidence, provenance).

Required mechanisms to investigate:

• frozen independent first passes before reveal;
• deliberately heterogeneous initial hypotheses/methods;
• disjoint retrieval/source partitions where feasible;
• calibrated confidence (score later with Brier/log score);
• evidence-correlation matrix and effective diversity estimate;
• explicit unique-finding ledger;
• direct same-budget baseline and simple-vote baseline;
• small panel default (e.g. 3–5) unless marginal value justifies another seat;
• synthesis from evidence/claim graph, not authority or vote count;
• stopping based on marginal unique verified defect yield / estimated value of another seat.

Useful heuristic for exchangeably correlated judgments to study (not a guarantee):

n_eff ≈ n / (1 + (n-1)ρ)

This makes the design consequence concrete: five highly correlated agents may provide little more effective independent information than one or two.

Research anchors:

• Choi, Zhu & Li, Debate or Vote? (NeurIPS 2025): majority voting accounts for much of vanilla multi-agent-debate gain; debate alone is not automatically corrective.
• Pitre, Ramakrishnan & Wang, CONSENSAGENT (ACL Findings 2025): sycophancy is a measurable failure mode in multi-agent debate.
• Zhu et al., Demystifying Multi-Agent Debate: The Role of Confidence and Diversity (ACL Findings 2026): diversity of initial hypotheses and calibrated confidence are key mechanisms.
• Sunstein & Hastie, Four Failures of Deliberating Groups: cascades, polarization, amplification, and shared-information dominance.
• Kao & Couzin (2014): correlated information can make a finite/small group more accurate than ever-larger groups.

Open-source lead:

• deeplearning-wisc/debate-or-vote — reproduce vote/debate baselines rather than building them from scratch.

4. Five Gems

The five Gems remain separate because their epistemic obligations are genuinely different. The main weakness is that most are currently specifications without dedicated tool adapters/receipts.

Mind / Formal Reasoning

Current spec correctly demands claim-native verification, but does not provide a standard obligation schema or execute solver/prover adapters.

Investigate adapters for:

• exact Python/SymPy calculations;
• Z3/SMT for satisfiability/countermodels;
• Lean 4 / Coq for machine-checked proofs where appropriate;
• proof receipts containing input/formalization hash, tool/version, output/proof object, and scope.

Research lead: Rango (2024) shows adaptive retrieval of relevant premises/proofs materially improves automated Coq proving; proof engineering literature should guide integration rather than treating a prover as a magic oracle.

Space / Research Discovery

Current tools/scout.py is only a keyless OpenAlex lookup. It does not implement the skill’s full discovery contract: multi-index search, deduplication, seed-set expansion, forward/backward citation chasing, inclusion/exclusion ledger, coverage/stopping estimate, or source-independence tracking.

Investigate:

• asreview/asreview for active-learning-assisted high-recall screening;
• nealhaddaway/citationchaser for forward/backward citation chasing;
• OpenAlex + Crossref + OpenCitations + other allowed indexes;
• explicit query/search log and saturation/stopping state.

Reality / Method Synthesis

Currently specification-only. Implement a machine-readable candidate object:

• named gap/constraint;
• changed assumption/mechanism;
• nearest prior art + delta;
• inputs/outputs/invariants;
• predicted failure modes;
• negative control;
• transfer target;
• ablation/verifier plan.

Research to investigate: quality-diversity / novelty search, morphological design, analogical retrieval, and program synthesis — but only mechanisms that improve candidate diversity/coverage under a fixed verification budget should be admitted.

Power / Engineering Verification

Current skill asks for the right things, but there is no generic verifier orchestration layer that chooses checks based on the claim/failure model.

Tool leads:

• HypothesisWorks/hypothesis — property-based testing;
• boxed/mutmut — mutation testing to test whether the test suite detects plausible faults;
• semgrep/semgrep / CodeQL — static/security analysis;
• symbolic/model/formal tools where justified by the claim.

Add executable receipts and explicitly measure which defect classes each verifier can/cannot observe.

Time / Evaluation & Benchmarking

Current benchmark protocols are project-specific and good on freezing/exclusions, but the Gem lacks a reusable statistics/evaluation engine.

Implement reusable paired-analysis primitives:

• paired binary discordance table + appropriate McNemar/exact-unconditional or preregistered paired test;
• Wilson/Jeffreys intervals for accuracy rather than Wald intervals;
• paired bootstrap/permutation for continuous metrics;
• contamination/exclusion ledger as first-class data;
• power/minimum-detectable-effect planning;
• confidence sequences/e-processes if experiments will be monitored/adaptively stopped;
• multiplicity control across benchmarks/variants.

Open-source/statistical leads:

• gostevehoward/confseq — time-uniform confidence sequences;
• jakorostami/expectation — e-values/e-processes/sequential tests (pre-release; evaluate before depending on it).

Cross-cutting implementation rule

Every component should eventually expose the same four layers:

1. SPEC — what obligation the component owns.
2. STATE — typed machine-readable state/events.
3. ACTION/TOOL — what actual evidence-producing method runs.
4. RECEIPT — what was observed, verifier/tool identity, hashes/provenance, uncertainty and unresolved state.

This provides a clean boundary for the later empirical phase: compare the complete implemented mechanism against same-model/same-tool/same-budget controls without changing the mechanism after seeing benchmark answers.

Before empirical benches

• Deep research each section above using primary papers/systematic reviews plus GitHub implementation inspection.
• Write explicit formal/algorithmic specs for the proposed mechanisms.
• Build unit + mutation/hazard tests for mechanical behavior only.
• Freeze implementation.
• Then preregister empirical component ablations and complete-system benches.