Skip to content

Navigation Menu

Sign in
Sign up

fix(eval): drive the agent eval with the OAuth token, and probe it for real #3403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kovtcharov-amd wants to merge 6 commits into main
base: main
Choose a base branch
Loading
from fix/eval-gate-oauth
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions .github/workflows/test_eval_agent_gemma_consolidation.yml
View file Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,11 @@ jobs:
# build the backend-default agent these scenarios run against, so install
# the LOCAL editable hub package or every scenario dies at session
# creation. Core gaia stays editable so eval fixtures resolve in-repo.
install-package: '-e .[dev,eval,ui,api] -e hub/agents/chat/python'
# `mcp` is not optional for this gate: every scenario is driven through
# the Agent UI MCP server, and without the extra that server exits with
# ModuleNotFoundError before writing a protocol frame — which the client
# reports only as CONNECTION_CLOSED.
install-package: '-e .[dev,eval,ui,api,mcp] -e hub/agents/chat/python'

# `gaia eval agent` does not merely call the Anthropic API - it shells out
# to `claude -p` with an MCP config to DRIVE each scenario (runner.py:937),
Expand Down Expand Up @@ -384,7 +388,11 @@ jobs:

- name: Preflight — judge key, baselines, Lemonade
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Blank when the OAuth token is present, deliberately: runner.py adds
# `--bare` only when ANTHROPIC_API_KEY is set, and `--bare` restricts
# auth to the key alone, so leaving both set would ignore the token.
ANTHROPIC_API_KEY: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN == '' && secrets.ANTHROPIC_API_KEY || '' }}
run: |
$ErrorActionPreference = "Stop"
$ProgressPreference = "SilentlyContinue"
Expand Down Expand Up @@ -496,11 +504,22 @@ jobs:
}


if (-not $env:ANTHROPIC_API_KEY) {
Write-Host "::error::ANTHROPIC_API_KEY is not set. The eval judge (src/gaia/eval/claude.py) reads it from the environment; without it every scenario scores as an infra error and the scorecard is meaningless. Add the ANTHROPIC_API_KEY repository secret, or run the eval locally on AMD hardware."
if (-not $env:CLAUDE_CODE_OAUTH_TOKEN -and -not $env:ANTHROPIC_API_KEY) {
Write-Host "::error::Neither CLAUDE_CODE_OAUTH_TOKEN nor ANTHROPIC_API_KEY is set. `gaia eval agent` drives AND scores each scenario through ``claude -p`` (src/gaia/eval/runner.py), so without a credential every scenario errors in a few seconds and the scorecard is meaningless. Add either repository secret, or run the eval locally on AMD hardware."
exit 1
}

# Assert the credential is ACCEPTED, not merely present. #3341 sat red
# for weeks because the key existed and the account behind it was out
# of credit: the preflight passed and all five scenarios then died in
# three seconds each with their stdout discarded.
claude -p --model $env:EVAL_MODEL "reply with: ok" 2>&1 | Out-String -OutVariable probe | Out-Null
if ($LASTEXITCODE -ne 0) {
Write-Host "::error::The Claude credential is present but a one-line probe was rejected, so every scenario would error. Response: $probe"
exit 1
}
Write-Host "Claude credential accepted by a live probe."

foreach ($category in @("rag_quality", "context_retention", "tool_selection")) {
$baseline = Join-Path $env:BASELINE_DIR "scorecard_$category.json"
if (-not (Test-Path $baseline)) {
Expand Down Expand Up @@ -785,7 +804,11 @@ jobs:
# derivation on the job above. Same convention as email_scorecard_refresh.yml.
timeout-minutes: 400
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Blank when the OAuth token is present, deliberately: runner.py adds
# `--bare` only when ANTHROPIC_API_KEY is set, and `--bare` restricts
# auth to the key alone, so leaving both set would ignore the token.
ANTHROPIC_API_KEY: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN == '' && secrets.ANTHROPIC_API_KEY || '' }}
run: |
# Deliberately NOT "Stop": the eval's stderr is piped through `2>&1`
# below, and under Stop the first stderr line from a native command
Expand Down
4 changes: 2 additions & 2 deletions eval/mcp-config.json
View file Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"mcpServers": {
"gaia-agent-ui": {
"command": "uv",
"args": ["run", "python", "-m", "gaia.mcp.servers.agent_ui_mcp", "--stdio"],
"command": "python",
"args": ["eval/mcp_server_launcher.py", "--stdio"],
"env": {
"GAIA_MEMORY_MCP_ALWAYS": "1",
"GAIA_MEMORY_ADMIN": "1"
Expand Down
92 changes: 92 additions & 0 deletions eval/mcp_server_launcher.py
View file Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
#!/usr/bin/env python
# Copyright(C) 2026 Advanced Micro Devices, Inc. All rights reserved.
# SPDX-License-Identifier: MIT
"""Launch the Agent UI MCP server with its stderr preserved.

The eval spawns this server once per scenario, as a grandchild: the runner
starts ``claude -p``, and ``claude -p`` starts the server. When the server dies
the client reports only ``CONNECTION_CLOSED``, and the server's own stderr goes
nowhere — it is not the scenario subprocess, so capturing *that* process's
output (#3375) does not reach it either.

Three runs of the eval gate were spent inferring a cause from timings that one
line of this log would have stated outright. So: exec the real server in-process
with stderr tee'd to a file the workflow uploads.

stdout is untouched and unbuffered. It carries the MCP protocol, and a single
stray byte on it desynchronises the client — which is why the diagnostics go to
a file rather than to stderr-as-console or, worse, to stdout.
"""

from __future__ import annotations

import os
import runpy
import sys
import traceback
from datetime import datetime, timezone
from pathlib import Path

#: Where to tee stderr. The workflow uploads this directory as an artifact.
_LOG_DIR = Path(os.environ.get("GAIA_MCP_LOG_DIR", "eval-out"))
_LOG_PATH = _LOG_DIR / "mcp-server.err.log"


class _Tee:
"""Write to both the real stderr and the log, so neither is lost."""

def __init__(self, *streams):
self._streams = streams

def write(self, data):
for stream in self._streams:
try:
stream.write(data)
stream.flush()
except Exception: # noqa: BLE001 - a broken tee must not kill the server
pass
return len(data)

def flush(self):
for stream in self._streams:
try:
stream.flush()
except Exception: # noqa: BLE001
pass


def main() -> int:
try:
_LOG_DIR.mkdir(parents=True, exist_ok=True)
log = open(_LOG_PATH, "a", encoding="utf-8", errors="replace")
except OSError:
# A log we cannot open must not stop the server from starting.
log = None

if log is not None:
sys.stderr = _Tee(sys.stderr, log)
stamp = datetime.now(timezone.utc).isoformat(timespec="seconds")
print(
f"\n=== MCP server launch {stamp} pid={os.getpid()} "
f"python={sys.executable} ===",
file=sys.stderr,
)

# argv[0] must look like the module's own invocation, and the --stdio flag
# the config passes has to survive.
sys.argv = ["gaia.mcp.servers.agent_ui_mcp", *sys.argv[1:]]
try:
runpy.run_module("gaia.mcp.servers.agent_ui_mcp", run_name="__main__")
except SystemExit as exc:
code = exc.code if isinstance(exc.code, int) else 0
print(f"=== MCP server exited with {code} ===", file=sys.stderr)
return code
except BaseException: # noqa: BLE001 - the whole point is to record it
print("=== MCP server raised ===", file=sys.stderr)
traceback.print_exc(file=sys.stderr)
return 1
return 0


if __name__ == "__main__":
sys.exit(main())
Loading

AltStyle によって変換されたページ (->オリジナル) /