-
Notifications
You must be signed in to change notification settings - Fork 162
feat(eval): step-level benchmark to measure the substrate work #3388
Description
Every other issue in this milestone claims the agent will get better at running a real
development task. Nothing currently measures whether it does. The existing evals score
conversational answers, so a substrate change — a persistent shell, a verification state,
parallel dispatch — can land with all evals green and no evidence it helped.
This adds a step-level benchmark: freeze real decision points, show the agent only what it
knew at that moment, and score the action it proposes. It answers "at step N, given this
state, did the harness choose well", which is the question this milestone is actually
about.
The approach is already prototyped and produced the numbers cited across these issues.
Running it against GAIA today gives a baseline of 1.58/3 mean action quality versus
2.77/3 for the reference harness on identical inputs, same model behind both. That gap
is the milestone's headroom, and it is what should shrink as the substrate work lands.
Technical details
Unit. One record is one decision point — a single inference call that dispatched at
least one tool. Not one tool call: keying on tool calls destroys the ability to see whether
the agent fanned out.
What the agent is shown: goal, instruction, recent steps with outcomes, known paths,
observed binaries, files already read. Withheld: the action, its result, and all
grading metadata.
Scoring, three layers:
- Deterministic agreement — tool match, and static argument checks (does the path exist in
what the agent has seen, does the pattern compile, is the binary available). - Every record carries the reference's own result on each check. A check is only counted
where the reference passed it; otherwise it cannot discriminate and is dropped. Without
this calibration the checks are wrong 30-47% of the time. - An LLM judge scoring action quality 0-3, with a planted bad action in every comparison
as a validity control.
Two things the design must keep, both learned the hard way:
- Records where the reference action failed invert: reproducing a timeout scores zero.
Otherwise the benchmark rewards reproducing the worst habits in the corpus. - The tool list shown to the agent must be the agent's own tool surface. Passing the
reference harness's tool names through made a first run invalid, with the agent naming
tools it does not have.
Reporting rules. Per capability axis and per use-case, never one aggregate number,
never called "accuracy" (it is agreement with one strong harness, not ground truth), and
never called task success — no transcript records whether the goal was met.
Cost. ~60 records is a few minutes and roughly 5ドル API-equivalent. Prompt caching is not
currently engaged and would cut the dominant input cost substantially, since the system
prompt and tool schemas ride on every call.
The corpus the records derive from is internal and not public; the benchmark artifacts stay
out of this repository. Only the builder and grader code would be contributed here.
Acceptance criteria
- Benchmark can be regenerated on a clean machine from documented steps
- Grader ships as code, so every consumer scores identically
- Replaying the reference action scores 100% — the calibration check that proves the
grader is sound before any candidate is measured - A deliberately bad action scores near zero
- Per-axis scorecard for GAIA committed as a baseline, with the two polarities separate
- Runnable against GAIA in CI or on demand, so a substrate PR can show its effect