-
Notifications
You must be signed in to change notification settings - Fork 162
feat(agent): materialize a project map at task start - #3404
feat(agent): materialize a project map at task start #3404kovtcharov-amd wants to merge 2 commits into
Conversation
The flagship opened every task blind — it guessed directory names, guessed which commands existed, and guessed which shell it was talking to, and each wrong guess cost a full round trip to learn something one orientation pass establishes once. It now gets a 600-token block in the system prompt naming the project root, the directory shape two levels deep, the likely entry points, which commands are installed (and which are not), the three platform differences that change command syntax, and whether the semantic code index is built. The binary half extends the existing day-0 system-context probe rather than adding a second one: `probe_binaries` is now the single PATH probe, shared by `collect_system_info` and the map, and `DEV_TOOL_PROBES` widens the seven desktop-app markers to the developer toolchain. `index_codebase` gains its first automatic trigger. When the root satisfies `is_code_repository` — a VCS directory or a recognised manifest, non-recursive — and no index exists, the map starts one in a background thread and says so in the prompt. `GAIA_PROJECT_MAP_AUTO_INDEX=0` turns it off. Budget is 600 tokens, 1.8% of the 32K NPU window, enforced by `render_project_map` on every render and pinned by a test against a 200-directory repository. The map is cached per root and invalidated by a fingerprint over the top-level listing, manifest contents, VCS head and PATH.
... itself Review follow-ups on the task-start project map, each a bug a user would have hit: - The index line parsed the whole code-index metadata — every chunk's text — on every prompt composition, several times a turn. `CodeIndexSDK.is_indexed` answers the same question with two `exists()` calls. - A background index that died left the prompt saying "building now" for the rest of the session, so the model kept waiting for something that would never arrive. The trigger is now a four-state machine and inspects the tool's JSON, which is how `index_codebase` reports a refusal rather than raising. - The map told the model `run_shell_command` accepts `uv`, `npm` and `python`. It accepts none of them. Installed-and-allowlisted, installed-but-refused, and not-installed are now three separate lines — and the whole section is omitted for an agent that has no shell tool. - In dev mode the daemon launches the agent sidecar from the GAIA checkout, so the working directory resolved to GAIA's own source and it would have background-indexed itself. `is_agent_own_source` rejects that; an explicit `GAIA_PROJECT_ROOT` is exempt. - Listing `ProjectMapMixin` after the base agent silently disabled the index trigger while the prompt still rendered. `__init_subclass__` now raises. Also: the root resolves once per session so the map and the code index cannot describe two different trees, and the day-0 memory fact that duplicated `git`/`node`/`docker` (and carried an `except Exception: pass`) is gone — `DEV_TOOL_PROBES` feeds the map, which is all #3379 asked for.
Skill audit
✅ All audited skills cleared the tier they claim. Per-finding detail is withheld here on purpose. Read it in the Security > Code scanning tab, or download the |
Verdict: Request changes
This adds a ~600-token orientation block to the agent's system prompt — directory shape, entry points, installed commands, platform quirks — plus a first automatic trigger for code indexing. The static half genuinely works and is well tested; the dynamic half does not, and the docs promise it in four places.
The map is built once and then frozen for the session. The agent's system prompt is only rebuilt when its tool or skill selection changes, so nothing re-renders the map when the project or the index state moves. The states this PR advertises — "building now in the background", "build FAILED" — never reach the model, and a directory created mid-session never shows up. That was observed on a real two-turn run, not inferred. Either refresh the prompt when the map's own content changes, or drop the changing-state language from the guide, the CHANGELOG and SPEC so the docs match what ships.
A project with GAIA installed into a venv inside it gets no map at all. The "don't map GAIA's own source" guard rejects any directory that contains the running gaia package — which includes a user's own repo the moment their virtualenv lives inside it. The agent then logs that the user's project is "GAIA's own source tree" and silently skips the feature. Narrow the check to an actual GAIA checkout.
A vanished project directory takes the whole turn down. If the project root is deleted, renamed or unmounted mid-session, the task-start hook raises and the query fails instead of just losing the map. The prompt path already degrades gracefully here; the hook should too.
Also worth noting before merge: this changes the system prompt, so CLAUDE.md requires an eval run. The PR is upfront that #3341 blocks it — please land the eval when that clears, since a new always-on prompt block is exactly the kind of change evals catch.
Real-world evidence
An evidence stage ran the gaia-testing skill on a no-inference ubuntu-latest runner and produced evidence-bundle.md. It confirms the map reaches a real prompt, and it is also where the freeze above was caught.
The block captured from an actual gaia-agent --dev turn against a scratch repo planted with unguessable names:
==== PROJECT MAP ====
Root: /tmp/zephyr-proj
Code repository: yes (.git, pyproject.toml, package.json, Makefile)
Platform (these three change the commands you write):
- Path separator: /
- Paths with spaces: wrap in single quotes: '/home/me/My Docs'
- Shell dialect: bash
Directories:
- docs/
- src/ (violet_otter_92)
- tests/ (harness)
Entry points: main.py, Makefile, npm run otter-lint, npm run zephyr-build
NOT installed, do not invoke: pnpm, poetry, powershell, rg, uv
run_shell_command accepts: arch, basename, cat, date, df, diff, dir, dirname, du, echo, file, find, free, git, grep, head, hostname, less, ls, lsblk, lscpu, lspci, lsusb, more, nproc, printf, ps, pwd, sort, stat, sysctl, tail, top, uname, uniq, uptime, wc, whereis, which, whoami
Code index: not built — call index_codebase to enable semantic code search
Measured at 247 of the 600-token budget. The planted names are all present, so it is read off the real tree. Root resolution also held on a real launch — from the GAIA checkout with no override it declined to map itself:
[project-map] /home/runner/work/gaia/gaia is GAIA's own source tree — no map. Set GAIA_PROJECT_ROOT to the project you want mapped.
The auto-index trigger fired and failed loudly when the embedding backend was down:
INFO | [project-map] indexing /tmp/zephyr-proj in the background ERROR | [project-map] background index of /tmp/zephyr-proj failed: Request failed: HTTPConnectionPool(host='localhost', port=13305) ... Connection refused
But the prompt never followed. Both turns of that run rendered the same line, after the failure:
$ grep -o "Code index: [^\\]*" ~/.gaia/logs/gaia-agent.log Code index: not built — call index_codebase to enable semantic code search Code index: not built — call index_codebase to enable semantic code search
And a directory created between turn 1 and turn 2 of one process never appeared — both maps byte-identical. That is the first blocking finding above, confirmed against the code.
The bundle also caught a black failure the PR introduces in the hub agent's agent.py (clean at the base commit) — python util/lint.py --black --fix clears it.
Adjacent surfaces were spot-checked green: /health, three memory routes, gaia memory status, gaia memory bootstrap --system (12 facts through the refactored binary probe), the same 12 via POST /api/memory/refresh-system-context, and the docs nav edit parsing with the new page referenced.
Deferred to the strix-halo lane and not covered by this verdict: the Agent UI screenshot, any real LLM turn, index completion (the "built — use search_code_index" line), gaia eval agent, and the Windows PlatformQuirks branch. For those surfaces this review rests on static reading alone.
🔍 Technical details
🟡 The rendered map is frozen after the first composition (src/gaia/agents/base/project_map.py:657)
get_project_map_system_prompt only runs inside _compose_system_prompt, and the composed prompt is cached in _system_prompt_cache. The only per-turn invalidation is _refresh_active_tool_filter → _apply_tool_filter, which recomputes only when the filter selection changes (agent.py:1288). With a stable filter the cache — and the map inside it — never moves. show_prompts composes it at construction (agent.py:888), i.e. before the first _on_task_start, which is why the run above rendered not built even on turn 2.
Adding the name to VOLATILE_PROMPT_FRAGMENTS doesn't help: that set only controls fragment ordering (agent.py:968), not recomputation.
The unit tests can't see this — test_prompt_says_building_while_the_index_is_running and test_prompt_reports_a_failed_index_instead_of_waiting_forever call get_project_map_system_prompt() directly, bypassing the cache entirely. A test that goes through the cached system_prompt property would have failed.
Fix options: call rebuild_system_prompt() from _on_task_start when the rendered fragment differs from last turn's, and again from the background thread's terminal states (_DONE / _FAILED). Cheap enough — the map is cached behind _fingerprint, so the recompose is a string join, not a walk. Whatever the fix, the claims in docs/guides/project-map.mdx ("Caching ... rebuilt when a fingerprint ... changes"), the CHANGELOG entry and SPEC.md should end up describing the shipped behaviour.
🟡 is_agent_own_source rejects any project that contains the installed gaia package (src/gaia/agents/base/project_map.py:416)
package = Path(gaia.__file__).resolve().parent path = Path(root).resolve() return path == package or path in package.parents
package.parents is every ancestor. For a user project with an in-tree virtualenv — /proj/.venv/lib/python3.12/site-packages/gaia — /proj is in package.parents, so resolve_project_root() returns None and logs that the user's repo is "GAIA's own source tree". That is a normal layout for anyone doing uv venv && uv pip install gaia inside their project, and it is precisely the developer-in-a-repo case the feature targets.
Tighten it to a GAIA checkout rather than any containing directory — e.g. require the package to sit at root/src/gaia or root/gaia, or bail out when a site-packages component separates root from the package:
import gaia
package = Path(gaia.__file__).resolve().parent
path = Path(root).resolve()
if path == package:
return True
# Only a checkout counts: an installed copy under the project's own venv
# (``<root>/.venv/lib/.../site-packages/gaia``) must not disqualify it.
return path in (package.parent, package.parent.parent) and (
"site-packages" not in package.parts
)
test_the_agents_own_source_tree_is_never_the_project passes either way (it uses parents[2] of the package, the checkout layout), so please add a case for the in-project-venv layout.
🟡 A raising _on_task_start fails the whole turn (src/gaia/agents/base/project_map.py:712)
_on_task_start → materialize_project_map → build_project_map → _fingerprint, whose bare os.scandir(root) raises if the root has been deleted, renamed, or unmounted since resolution (network share, external drive, git worktree remove). _process_query_impl doesn't guard the call (agent.py:4505), so the query dies with a FileNotFoundError instead of losing an optimisation.
Note the asymmetry: the prompt-fragment path is protected — _get_mixin_prompts catches and logs (agent.py:934). Orientation is a nicety; the hook should degrade the same way, e.g. wrap the body in except OSError and log at warning.
🟢 Nits
-
Unclosed
os.scandiriterators (project_map.py:262,:324,:344) —sorted(os.scandir(p))leaves the directory handle to the GC and emits aResourceWarningunder-W error.with os.scandir(p) as it: sorted(it, ...)is a one-line change in all three spots. -
The guide's sample map can't be produced by the renderer (
docs/guides/project-map.mdx:24) — it showsCode repository: yes (pyproject.toml, setup.py, package.json)for a Windows GAIA checkout, butrender_project_mapprefixes the VCS name, so a real checkout renders.git, pyproject.toml, .... The evidence bundle's captured block shows the true shape. -
_tool_names()returns the registry, not names (project_map.py:668) — annotatedDict[str, Any]and used for membership tests, so it works, but the name reads like it returns a name collection._tool_registry()would say what it is. -
A typo'd off-switch value silently disables auto-index (
project_map.py:592) —GAIA_PROJECT_MAP_AUTO_INDEX=tureis falsy and turns the feature off with no signal. Given the repo's fail-loudly rule, an unrecognised value is worth a warning. -
probe_binariesdrops the module's stated try/except discipline (system_context.py:81) —system_context.py's header says every collection step is wrapped so partial collection is fine; the new probe replaces a guardedshutil.whichloop with an unguarded one. Low risk, but it's a documented invariant of that module.
Strengths
- The evidence bundle is the model of what this rubric asks for. It didn't just show the happy path — it drove the real
gaia-agentprocess, planted unguessable names to prove the map is read rather than guessed, and reported two behaviours that didn't hold as observations rather than burying them. Both turned out to be real. - The MRO guard is the right kind of defensive design (
project_map.py:620). A wrong base order would produce a map that renders fine and silently never triggers indexing — the__init_subclass__check turns a symptomless bug into a class-definition error, andtest_a_wrong_base_order_fails_at_class_definitionpins it. is_indexed()as a presence check rather thanget_status()(code_index/sdk.py:512), with a test assertingget_statusis never called on the prompt path. That is the difference between a cheap per-turn line and parsing tens of megabytes of metadata.- The binary probe was unified rather than duplicated —
collect_system_infoand the map now share oneprobe_binarieswith one PATH-keyed cache, and the map keeps "installed" separate from "the shell tool will accept it", which is the distinction that actually prevents a wasted call.
The agent used to open every task blind: it guessed directory names, guessed which programs were installed, and guessed which shell it was talking to, and a meaningful share of its wasted steps were those guesses coming back as "no such file" or "command not found" — each a full round trip to learn something one orientation pass establishes once. Inside a code repository it now starts every task with a 600-token project map in the system prompt: the root, the directory shape two levels deep, the likely entry points, which commands are installed and which of those the shell tool will actually accept, the three platform differences that change command syntax, and whether the semantic code index is built.
index_codebasealso gains its first automatic trigger — until now nothing but the model deciding to call it ever built an index.Closes #3379.
The four numbers and predicates the issue asked to be named
is_code_repository(path)— a directory fromVCS_DIRS(.git,.hg,.svn) or a file fromPROJECT_MANIFESTS(14 entries) at the root. Non-recursive, so a home directory full of repositories is not itself one. Every manifest in the list is parametrized in the tests.PlatformQuirks— path separator, path quoting for spaces, shell dialect. A test asserts the dataclass has those three fields and no others, so the list stays closed.probe_binaries()is now the single PATH probe in the codebase;collect_system_infoand the map both go through it. The extension over the seven commands it already detected (git,code,cursor,node,docker,brew,npm— chosen because they double as desktop-app markers) isDEV_TOOL_PROBES: 29 build, package, runtime and VCS binaries. The map also crosses the result withrun_shell_command's own allowlist, so it can say up front which commands the shell tool will refuse instead of letting the agent discover that one refusal at a time.Two behaviours worth a reviewer's attention
Auto-indexing costs something on first contact. When the root is a repository with no index, the map starts
index_codebasein a background thread. On a large repo that is minutes of local embedding, and the embedder can evict the resident chat model — the same trade the RAG warm-up already makes, so the cost is first-turn latency, not a wrong answer.GAIA_PROJECT_MAP_AUTO_INDEX=0turns it off. It fires at most once per session, and a failure is reported in the prompt asbuild FAILEDrather than leaving the model waiting on something that will never arrive.Root resolution refuses to point at GAIA itself. In dev mode the daemon launches the agent sidecar with its working directory set to the GAIA checkout, so a naive cwd rule would have had the flagship map — and background-index — its own source tree.
is_agent_own_sourcerejects that. An explicitGAIA_PROJECT_ROOTis exempt: pointing GAIA at GAIA is legitimate when you mean it.Eval not run: this changes the system prompt, which CLAUDE.md flags as eval-affecting. #3341 tracks
gaia eval agentfailing repo-wide because the API account behindANTHROPIC_API_KEYis out of credit.Test plan
python -m pytest tests/unit/test_project_map.py -q— 64 tests: the predicate against all 14 manifests, the budget on a ×ばつ20-directory repo, cache invalidation on directory/manifest/PATH change, the four index-trigger states, and the MRO guard.python -m pytest tests/unit/ -q— no new failures against the pre-change baseline (616 failed / 9,621 passed before, 614 / 9,679 after; the deltas are the new tests). The email- and hub-agent modules fail identically before and after for an unrelated editable-install reason.python util/lint.py --all— black, isort, pylint, flake8, bandit and the agent-convention checks pass.GAIA_PROJECT_ROOT=$(pwd) python -c "from gaia.agents.base.project_map import *; from gaia.agents.base.turn_metrics import count_tokens; t=render_project_map(build_project_map(resolve_project_root())); print(t); print(count_tokens(t))"Expect ~470 tokens, and every name on the
run_shell_command accepts:line present inALLOWED_COMMANDS.resolve_project_root()returnsNoneand the prompt fragment is empty.