feat(agent): state whether an answer was verified, on every exit path - #3397

🔴 Critical — the command classifier matches anywhere in the string (src/gaia/agents/base/verification.py:42)

_CHECK_COMMAND_RE is applied with .search() over the whole command, so any occurrence of a runner token counts. Combined with _is_error_result (which reads return_code), a zero-exit pip install ruff, cat pytest.ini, grep -rn "pytest" src/ or git commit -m "fix mypy errors" produces Verification: verified — ruff ran and passed. The module docstring states the opposite intent ("a false positive would claim a check that never ran"), and test_command_classification only passes because its negative case is git commit -m 'wip'.

Anchor the match to a command position — start of string or immediately after a shell separator:

_CHECK_COMMAND_RE = re.compile(
 r"(?:\A|[;&|]\s*|\b(?:and|then)\s+)\s*(?:[\w./\\-]*[/\\])?("
 r"pytest|py\.test|tox|nox"
 r"|python\s+-m\s+(?:pytest|unittest)"
 r"|npm\s+(?:run\s+)?(?:test|lint|build|typecheck)"
 r"|yarn\s+(?:test|lint|build)"
 r"|pnpm\s+(?:run\s+)?(?:test|lint|build)"
 r"|go\s+(?:test|vet|build)"
 r"|cargo\s+(?:test|clippy|check|build)"
 r"|dotnet\s+(?:test|build)"
 r"|mvn\s+(?:test|verify)"
 r"|make\s+(?:test|check|lint|build)"
 r"|ctest|jest|vitest|mocha"
 r"|ruff|flake8|pylint|mypy|pyright|eslint|tsc|shellcheck"
 r"|python\s+util[/\\]lint\.py"
 r")\b",
 re.IGNORECASE,
)

I ran that variant against the file's own parametrize table: every existing positive still classifies, every false positive above returns None. One expectation shifts — python util/lint.py --all labels as python util/lint.py rather than util/lint.py, so that row needs updating. Please also add the confirmed false positives as negative cases in tests/unit/agents/test_verification_scope.py::test_command_classification — that test is what should have caught this.

🟡 Applies unconditionally to every agent and every surface (src/gaia/agents/base/agent.py:6483, :6590, :6606)

_with_verification_scope sits on the base loop, so EmailTriageAgent, the chat/doc profiles and the flagship all get "none of them a test, lint, or build" appended to answers where that vocabulary has no meaning — a RAG-grounded document answer reads as unchecked because retrieval is not a build step. It also reaches src/gaia/api/openai_server.py:364 (OpenAI-compatible completions) and the Telegram adapter, so third-party clients see an extra trailing line on every completion with no way to turn it off. Suggest a class-level opt-in (e.g. VERIFICATION_SCOPE_ENABLED, default on only for agents with a shell/test surface), or vocabulary that generalises beyond code work.

🟡 Appended without stripping an existing line (src/gaia/agents/base/agent.py:4476)

The PR notes the line rides in conversation history. Small local models imitate trailing patterns, and a model-emitted Verification: verified ... line is not removed before the real one is appended: the answer then carries two, strip_verification_scope (anchored to \Z) only removes the trailing one, and both SSEOutputHandler.print_final_answer and the email agent's grounded-answer comparison assume exactly one. Cheap fix:

 if not answer or not answer.strip():
 return answer
 answer = strip_verification_scope(answer).rstrip()
 return f"{answer}\n\n{self.verification_scope_statement()}"

(with strip_verification_scope added to the existing import).

🟢 Nits

• Doc drift (docs/sdk/core/agent-system.mdx:32): the tool list omits lint and run_test_suite, both of which are in _CHECK_TOOLS.
• Truncation can drop the verdict (verification.py:139): with three long labels the 200-char cut lands before "ran and passed", leaving a statement that names checks but no outcome. Truncating the label list rather than the whole string would keep the clause.
• Email-agent ordering (hub/agents/email/python/gaia_agent_email/agent.py:1230): on the direct-set exit paths ground_final_answer runs after the line is appended and can append its own correction below it, leaving the scope line mid-answer where the strip helper and the SSE extraction no longer see it.
• BuilderAgent runs its own loop and calls _execute_tool directly (src/gaia/agents/builder/agent.py:374), so its answers carry no statement at all — worth a note if the intent is universal coverage.

Strengths

• Hooking _note_verification_signal into _execute_tool_timed is the right seam: legacy, native tool-calling and forced-call branches are all covered without touching three call sites, and nested tool calls still record.
• Applying the line after finalize_answer, so a subclass rewrite cannot silently drop it, is a good call — and it has a test.
• The direct-set exit paths (cancel timeout, connection error, context overflow, parse give-up, loop break, max steps) each get their own test through the real loop with a stubbed client; that enumeration is the hard part of the issue and it is done carefully.
• Keeping the console-cancellation path empty, and testing that it stays empty, avoids turning a cancelled turn into a completed-looking one.