fix(eval): drive the agent eval with the OAuth token, and probe it for real - #3403

Verdict: Request changes

Switching the eval gate to the OAuth token is the right fix and the reasoning behind blanking the API key checks out against the code it cites. The new live probe, though, is written in a way that breaks exactly when it is supposed to work.

The blocking issue — the probe can't print its own error message. This step runs under PowerShell's "stop on any error" mode, and the probe captures the CLI's error output. In Windows PowerShell that combination makes the very first line of error output kill the step outright, so a rejected credential ends the run with a raw PowerShell stack trace instead of the actionable message the PR adds one line below it. The same step is also now fragile in the other direction: if the CLI prints any harmless warning to its error stream on a successful call, the preflight goes red for no reason — trading one always-red gate for another. This workflow already solves this exact problem elsewhere in the file, so the fix is a few lines and follows an established local pattern.

Worth calling out that the PR's own test plan item — "temporarily point it at a bogus token to check the error is the one a reader can act on" — is the check that would have caught this. Running it before merge is the whole verification here.

Real-world evidence

N/A — CI-workflow-only change; no evidence bundle was produced for this PR and this surface can't be exercised from the review lane. My verdict rests on static review plus reading the credential-selection code the PR's rationale depends on. The only meaningful proof is a dispatch of this workflow on the branch, and all three test-plan boxes are still unchecked — including the bogus-token check that targets the issue above.

 # Assert the credential is ACCEPTED, not merely present. #3341 sat red
 # for weeks because the key existed and the account behind it was out
 # of credit: the preflight passed and all five scenarios then died in
 # three seconds each with their stdout discarded.
 # Continue, not Stop: under Stop the first stderr line from `claude`
 # becomes a terminating NativeCommandError and the ::error:: below
 # never prints. Same reason as the eval step.
 $ErrorActionPreference = "Continue"
 $global:LASTEXITCODE = 0
 $probe = (& claude -p --model $env:EVAL_MODEL "reply with: ok" 2>&1 | Out-String)
 $probeCode = $LASTEXITCODE
 $ErrorActionPreference = "Stop"
 if ($probeCode -ne 0 -or -not $probe.Trim()) {
 Write-Host "::error::The Claude credential is present but a one-line probe was rejected, so every scenario would error. Exit $probeCode. Response: $probe"
 exit 1
 }
 Write-Host "Claude credential accepted by a live probe."

 Write-Host "::error::Neither CLAUDE_CODE_OAUTH_TOKEN nor ANTHROPIC_API_KEY is set. ``gaia eval agent`` drives AND scores each scenario through ``claude -p`` (src/gaia/eval/runner.py), so without a credential every scenario errors in a few seconds and the scorecard is meaningless. Add either repository secret, or run the eval locally on AMD hardware."

Verified on this PR's own run, and the credential half is fixed.

Claude credential accepted by a live probe.
 CLAUDE_CODE_OAUTH_TOKEN: ***
 ANTHROPIC_API_KEY:

The scenarios now reach the agent instead of dying at the door. Before, all five errored in 3 seconds; now they run 14–18s, and multi_step_plan ran for 993 seconds before timing out — that is a scenario genuinely executing, in the 200–680s band #3341 recorded for real runs.

The gate is still red, on a different and deeper cause. Four scenarios return INFRA_ERROR at 14–18s and one times out. The backend comes up healthy first, so this is past auth and past startup.

And the new failure is still invisible for the same reason the old one was. The runner prints the subprocess's stderr while claude -p reports on stdout, so all we can see is the shape. That makes #3375 the critical path now rather than a nice-to-have — until it lands, the next person gets to guess from timings the way this one did.

I would still merge this: it removes one real blocker, it is what every other Claude workflow in the repo already does, and the live probe means the next credential failure announces itself in the preflight instead of thirty minutes later as five mystery errors.

Second fix pushed, for the cause the first one exposed.

With the credential working, the scenarios got far enough to say what was actually wrong — the trace artifacts carry the agent's own diagnosis, identical across all four:

The gaia-agent-ui MCP server failed to connect (CONNECTION_CLOSED) before Phase 1 could begin. system_status() was not callable, so no session could be created and no turns were executed.

The eval's MCP config launched that server with uv run, which resolves the project environment rather than the active one. On a checkout without a synced .venv it creates an empty one, and the server dies before writing a single protocol frame. A stdio server that exits is CONNECTION_CLOSED to the client.

Reproduced directly rather than inferred:

$ uv run python -m gaia.mcp.servers.agent_ui_mcp --stdio
Creating virtual environment at: .venv
...Error while finding module specification for 'gaia.mcp.servers.agent_ui_mcp'
(ModuleNotFoundError: No module named 'gaia') # exit 1

It now launches with python, inheriting the environment the eval is already running in — the one that has GAIA installed.

That also explains why this looked environment-specific and was not: the config is checked in, so every runner and every branch hit it the same way.

Third run, and the gate is measuring for the first time. Not green yet, and the residual is a different shape of problem.

Two scenarios ran end to end and produced real scores — 9.93 PASS and 3.08 FAIL — where every scenario previously died before Phase 1. The 3.08 is a genuine product finding rather than infrastructure: the agent reports that search_file excludes .md from its default scope and never scans the eval corpus directory. That is exactly the kind of signal this gate exists to produce and has not produced once in weeks.

Three still report CONNECTION_CLOSED. So the uv run fix was necessary but not sufficient: the MCP server now starts, and sometimes stays up. Intermittent rather than absolute points at a startup race or contention on the runner, not a config error — a checked-in config fails identically every time, and this no longer does.

Where that leaves things, honestly:

• The two fixes here are both verified and worth merging on their own. Auth was genuinely broken; the uv run launch was genuinely broken; each was invisible behind the other.
• The remaining flake needs someone who can watch the runner while it happens. fix(eval): capture stdout when a scenario subprocess fails #3375 would help — the MCP server's own stderr is still discarded, so all anyone can see is that the connection closed, not why.
• Until this merges, the other five open PRs keep showing the same red gate, because the fix has to be on their branch or on main.

I have not merged it; that is a call for a human.

Correction: I pushed a third fix on a warm-up hypothesis, the next run contradicted it, and I have reverted it. Recording that here so nobody re-derives it.

The hypothesis was that the MCP server's first launches were slow rather than broken — scenarios 1-3 had failed at 19s/17s/36s while 4 and 5 completed in 197s and 336s, which reads as a warm-up curve. So I pre-warmed the import and set MCP_TIMEOUT=120000.

The next run failed all five at 15-22s. That is the disproof: had the wider startup window been in effect, a slow server would have been given 120 seconds before being dropped. Failing at the same ~15-20s means the server is exiting, not timing out — and the run where two scenarios passed was variance, not warming.

So this PR is back to the two fixes that are each independently reproduced:

1. Authenticate with CLAUDE_CODE_OAUTH_TOKEN — verified by the live probe passing and by scenarios reaching the agent at all.
2. Launch the MCP server with python rather than uv run — reproduced directly on this machine, where uv run creates an empty venv and exits with ModuleNotFoundError: No module named 'gaia'.

What is left is a server that starts and then exits, and its stderr is discarded, so there is nothing to read. That is #3375's fix, and it is now the blocker rather than a convenience — I have spent three runs inferring from timings what one line of captured stderr would have said outright.

Root cause found, and it is one line.

=== MCP server launch 2026年09月05日T15:55:42+00:00
 python=C:\actions-runner\_work\gaia\gaia\.venv\Scripts\python.exe ===
ModuleNotFoundError: No module named 'mcp'

The job installed -e .[dev,eval,ui,api]. mcp is an optional extra and was not in that list — while every scenario in this gate is driven through the Agent UI MCP server. The server exited before writing a protocol frame, and the client reported that as CONNECTION_CLOSED.

That is why it failed identically on main and on every PR, why re-running never helped, and why it looked like a runner fault: a missing dependency is perfectly deterministic.

Four commits, each of which had to come before the next could be seen:

1. Authenticate with the OAuth token. The API account is out of credit. Until this, nothing ran at all.
2. Launch the MCP server with python, not uv run. uv run built an empty venv and the server died with a different ModuleNotFoundError, for gaia.
3. Keep the server's stderr. It is a grandchild — the runner starts claude -p, which starts the server — so nothing was holding its output. fix(eval): capture stdout when a scenario subprocess fails #3375 captures the scenario subprocess and would not have reached it.
4. Install the mcp extra. The actual fault, visible only once (3) was in place.

I also pushed and reverted a warm-up hypothesis in between; the next run disproved it and I took it out rather than leave a speculative change in the PR.

Worth keeping regardless of this gate: the preflight now proves the credential is accepted rather than merely present, and the launcher means the next MCP failure explains itself instead of costing five runs of inference.