-
Notifications
You must be signed in to change notification settings - Fork 162
feat(agent): GAIA never emits a parallel tool call, and the Claude path would drop it #3387
Description
GAIA takes exactly one action per step, always. Measured against a 60-decision-point
benchmark drawn from ~300 real agent sessions, GAIA's mean tools-per-step is 1.00 across
every single answer, against 1.38 for the reference harness on the identical inputs —
18 of those 59 steps genuinely wanted two or more tools dispatched together.
For a user this shows up as latency, not wrongness. The agent reads two files it already
knows it needs by making two round trips instead of one. At the corpus's p90 depth of 64
sequential steps per instruction, that compounds into a materially slower task and more
context burned re-sending the conversation on every extra round trip.
#944 fixed the loop rejecting parallel tool_calls and is closed. This is the other half:
the loop can now accept them, but the agent never produces them. Acceptance was the bug;
emission is the gap.
Technical details
Measured with the step-level harness (see companion issue) over 60 decision points, same
model (claude-sonnet-5) behind both harnesses so the comparison isolates the harness:
| reference | GAIA | |
|---|---|---|
| Mean tools per step | 1.38 | 1.00 |
| Width distribution | {1: 42, 2: 14, 3: 3, 4: 1} |
{1: 59} |
| Median substantive shell segments per command | 1 | 1 |
| Median argument length | 324 chars | 160 chars |
The distribution is the tell: not a single 2-wide response in 59. That is a structural
ceiling rather than a judgement the model is making, so no prompt change will move it.
Related but distinct: GAIA also does about half the work within a step (160 vs 324 median
argument chars). Composing larger single commands is a separate lever from dispatching
several tools at once; this issue is only the latter.
Corpus context for sizing: parallel dispatch is 13.5% of responses in the source corpus,
mean width 1.16. This is a real but bounded win — worth doing after the shell and
verification work in this milestone, not before.
Acceptance criteria
- The agent loop can emit and execute more than one tool call in a single step
- Results are returned to the model in a single observation, correctly ordered
- A partial failure (one of N calls errors) does not discard the successful results
- Mean tools-per-step on the 60-record benchmark rises above 1.0, with the width
distribution no longer{1: n} - Regression test asserting a 2-wide dispatch round-trips through the loop