-
Notifications
You must be signed in to change notification settings - Fork 162
fix(observability): turn metrics record that a tool failed, not why #3392
Description
When an agent run goes wrong, GAIA's own telemetry cannot tell you why. TurnRecorder
captures which tool ran, how long it took and whether it succeeded — but not what
arguments it was called with, and not what the error said. So a failed step reads as
read_file · ok=false · 0.2s, which is enough to know something broke and nothing at all
about the cause.
That gap showed up immediately when benchmarking the flagship. Across 40 decision points on
three models, 32 proposed actions would have failed outright. Classifying why — an invented
path, a tool called with no arguments, a command wrong for the state — required reconstructing
the arguments from outside GAIA. From GAIA's own logs the failures are indistinguishable
from each other.
The recorder is already well placed for this: it captures the rendered prompt, the model, the
tools offered that turn, and per-step timings. Three fields are missing, and they are the
three that carry the diagnosis.
🔍 Technical details
src/gaia/agents/base/turn_metrics.py, TurnRecorder.record_tool:
entry = { "step": step, "name": name, "at": _now(), "wall_s": round(wall_s, 4), "ok": bool(ok), }
Missing:
args— the arguments the tool was called with, truncated and scrubbed. Without
this you cannot tell a wrong path from a wrong flag from a malformed call.error— the error class and message whenok=False. Currently the boolean is all
that survives; the exception text is logged atdebuglevel, if at all, and never lands
in the turn record.result_chars(and ideally a head) — result size drives context growth and is the
cheapest signal for "this step blew up the prompt".
Already captured and worth keeping: rendered_prompt per LLM call, tool_names offered,
skills_active, and the per-step wall time split between model and tool.
Two related gaps found in the same pass:
- Dynamic tool selection is not logged. The flagship narrows 67 tools to ~26 per turn
(dynamic_tools,dynamic_tools_max), and nothing records which 26 were offered on a
given step. When the agent picks a poor tool, there is no way to tell whether the better
one was even on the list. - A turn where the model returned no tool call is not distinguishable from one that
returned a bad one. On a small local model this happened on 4 of 40 decision points; it
needs its own marker rather than looking like a silent no-op.
Scrubbing note: arguments and errors can contain absolute paths and pasted content, so the
recorder should apply the same redaction the diagnostics bundle uses before writing.
Acceptance criteria
-
record_toolcaptures scrubbed arguments, error class and message, and result size - The per-turn tool list actually offered to the model is recorded when dynamic
selection is on - A turn that produced no tool call is recorded as such, distinctly from a failed call
-
gaia stats(or the turn summary) can answer "which step failed, on what arguments,
with what error" without re-running anything - Redaction applied to arguments and error text before they are written