-
Notifications
You must be signed in to change notification settings - Fork 162
feat(shell): --bypass-permissions lifts the shell guardrails too - #3394
feat(shell): --bypass-permissions lifts the shell guardrails too #3394kovtcharov-amd wants to merge 4 commits into
Conversation
Running the unit suite could pop a Google OAuth consent screen on the developer's desktop, mid-run, with a dummy client id — so the window that stole focus was also an "Access blocked: invalid_client" error. connectors.flow.start_authorization launches the browser from a fire-and-forget asyncio.ensure_future task that resolves webbrowser.open when it runs, which can be after the test that patched it has finished and monkeypatch has restored the real function. Every connector test does patch the launcher; the patch just isn't guaranteed to still be in place at launch time, which is why the popup was intermittent. Blocking the launchers for the whole session closes the race from the other side: a per-test patch now restores to the stub, never to the real function.
Bypass permissions turned off the confirmation prompt and nothing else, so a user who had granted blanket consent still could not get the agent to run a build or a test suite. Compound commands were refused before they parsed, and no interpreter, test runner or package manager was in the allowlist — "verify your work before claiming it is done" was not something the agent could do, however well it was prompted. Under bypass, the shell's own gates now come off with the prompt: operators (&&, ||, ;, >) parse and run, the read-only binary policy is replaced by a developer set, and the rate limit is lifted. Off by default and byte-identical to before when off — ALLOWED_COMMANDS is untouched and the read-only tier keeps claiming exactly what it claims, so #2768's hardening of it stays meaningful. The developer set adds node, npm, make, cmake, go, cargo, sed, awk, curl, sleep, timeout, export, cp and mv. It also names python, python3, pytest and gh, which are consolidation rather than new reach: those already had paths via execute_python_file and shell:execute skill grants, and listing them here gives bypass one answer to "may this binary run" instead of three. rm stays out — not a boundary, since anything in the set can delete a file, but a tripwire against an accidental recursive delete. Stdio transport only. The HTTP surface is a bound socket, and an unguarded shell reachable over it is remote code execution rather than a relaxed permission model, so the handler pins bypass off and the request model forbids unknown fields. bypass_permissions is also a separate handler attribute from auto_approve_gated_tools: an unattended harness that merely pre-approves prompts must not inherit an unguarded shell. Skipping the prompt does not skip the record. Every command run under bypass is written to file_audit.log with its full arguments and its per-segment breakdown, so `cd build && make` is two auditable invocations rather than one opaque string. The per-segment walk is what produces that record, so it stays intact in both modes. Closes #3373 Closes #3374
Skill audit
✅ All audited skills cleared the tier they claim. Per-finding detail is withheld here on purpose. Read it in the Security > Code scanning tab, or download the |
Request changes
This extends --bypass-permissions so it also lifts the shell tool's own guardrails — compound commands parse, a developer binary set replaces the read-only allowlist, and the rate limit comes off — with the default path left alone. The design, the stdio-only reasoning and the test coverage are all strong. One gap is worth fixing before merge.
A command containing a line break slips past the segment walk. Under bypass, the tokenizer treats a newline as plain whitespace, so ls followed by a newline and rm -rf /tmp/x is checked as if it were a single ls invocation — it passes, and then the real shell runs both halves. Two things the PR promises quietly stop holding: the deliberate rm tripwire never fires, and the audit trail records one nonsense invocation instead of the two commands that actually ran. That matters most for the audit, because the shipped security doc tells operators that command substitution is the only thing bypass cannot audit precisely. Either treat a line break as a command separator, or refuse a multi-line command under bypass and say so.
Everything else is minor: the helper's own docstring names a gate that was never wired up, output redirection is silently swallowed for one narrow skill-granted case, and the PR quietly carries a second, unmentioned change (the test-suite browser guard).
No security escalation. The gap above sits entirely inside a mode the user deliberately turned on to get arbitrary code execution, and the default posture is unchanged — this is audit-integrity and a broken accident-tripwire, not an escalation path.
Real-world evidence
evidence-bundle.md reports the stage errored before producing anything (auth / install / rate-limit; see the run logs). The changed surface was not exercised here, and this verdict rests on static review alone. I could not run the test suite either (no pytest in this environment), so the "164 pass" claim is unverified from my side. The one thing I did execute is the tokenizer, reproduced standalone, which is what confirms the newline finding above.
The PR description's test plan names exactly the right runs — the refused-then-executed cd . && echo HI pair through the real stdio CLI, and the file_audit.log line — but every box is unchecked and no captured output is pasted in. Paste those two outputs into the description and the CLI surface is covered; the TUI banner//bypass wording still wants a look on the strix-halo lane.
🔍 Technical details
Issues
🟡 A newline is not a segment separator, so a multi-line command is validated as one command and executed as several (src/gaia/agents/tools/shell_tools.py:509, :512)
_tokenize sets whitespace_split = True and leaves \n in shlex's default whitespace, so it is dropped rather than emitted as a token, and _SEGMENT_SEPARATORS never sees it. Reproduced standalone:
'ls\nrm -rf /tmp/x' -> ['ls', 'rm', '-rf', '/tmp/x'] -> segments: ['ls']
'ls; rm -rf x' -> ['ls', ';', 'rm', '-rf', 'x'] -> segments: ['ls', 'rm'] # correct
_validate_bypass_command then only ever sees ls, returns None, and run_shell_command executes with shell=True on the original string (exec_cmd = command), so the shell runs both. Consequences:
rmis reachable under bypass without the refusal the set exists to produce. Explicitly not a security boundary per your own note — but it is the accident tripwire, and this is exactly the accident it was for._audit_shell_executionwritessegments=[["ls","rm","-rf","/tmp/x"]]— one invocation that never happened, instead of the two that did.docs/plans/security-model.mdx("Command substitution is the one thing bypass cannot audit precisely") andSPEC.mdboth become inaccurate.
Rewriting \n to ; before lexing would fix both but mis-splits a literal newline inside a quoted operand. The safe-fail version, if you'd rather not touch tokenisation:
bypass = self.bypass_gates_active()
if bypass and "\n" in command.strip():
return (
{
"status": "error",
"error": (
"Multi-line commands are not accepted, even with bypass "
"permissions active."
),
"has_errors": True,
"hint": (
"A line break is a command separator the per-segment "
"audit record cannot see. Join the commands with '&&' "
"or ';', or put the script in a file and run it."
),
},
[],
)
(goes in _validate_shell_command right after bypass = self.bypass_gates_active(), before the operator check.) Whichever route you pick, add a case to TestPerSegmentWalkSurvivesBypass — check("ls\nrm -rf /tmp/foo", bypass=True) is not None — since that class is precisely where a reader will look for this guarantee.
🟢 bypass_gates_active's docstring names a gate that doesn't read it (shell_tools.py:582)
Every gate reads this —
_validate_shell_command,skill_grant_covers_call, the tool description, the executor
The run_shell_command description is a static string (shell_tools.py:910) that still says "Pipes (|) are supported" and gives only read-only examples. So the model is never told the surface widened, which is the PR's stated goal ("verify your work before claiming it is done"). Either drop the clause from the docstring, or make the description bypass-aware — the latter is a tool-description change and would need an eval run per CLAUDE.md, so the docstring edit is the cheap correct fix if you want to keep the scope tight.
🟢 Redirection is silently swallowed for a lone skill-granted segment under bypass (shell_tools.py:1128)
> is not in _SEGMENT_SEPARATORS, so gh issue list > out.txt is one segment; with a shell:execute:gh grant active lone_granted_segment is true, use_shell stays false, and gh receives literal > and out.txt as argv. Narrow (needs both a grant and bypass), and keeping argv for granted binaries is deliberate and right — but the result is a wrong answer rather than a refusal. Worth either excluding redirect tokens from lone_granted_segment, or a comment saying redirection is intentionally inert on that path.
🟢 An unmentioned second thread rides along (tests/conftest.py:38, tests/unit/test_no_real_browser_launch.py)
The session-scoped browser guard is a good fix and well-explained in its own docstring, but it is unrelated to bypass permissions and gets no line in the PR description — a reviewer has to discover it from the diff. Worth a sentence in the description, or its own PR. Minor gap while you're there: it covers webbrowser.open/open_new/open_new_tab but not webbrowser.get(...).open(...), which is the other way start_authorization-style code reaches a real browser.
Strengths
- Every bypass test asserts both states.
TestDeveloperBinariesRefusedByDefaultandTestReadOnlySubGuardsLiftUnderBypassOnlypin the default tier alongside the bypass tier, so a binary leaking intoALLOWED_COMMANDSfails here rather than shipping.test_allowed_commands_carries_no_developer_binaryis the right invariant to have written down. - The tests go past validation into real execution.
TestExecutorUnderBypassspawns actual processes, which is what catches the two things validation-only tests cannot: that the operators reach a shell at all, and the lazily-created rate-limit deque that bypass skips creating.test_the_rate_limit_is_lifted's docstring names theAttributeErrorit exists to prevent. - The two-attribute split is the right call and is enforced.
bypass_permissionsseparate fromauto_approve_gated_tools, withtest_auto_approve_alone_does_not_lift_the_shell_gatespinning it, meansGAIA_AUTO_APPROVE_TOOLScan never quietly inherit an unguarded shell. - The HTTP transport is pinned shut from three directions — the handler assignment,
extra="forbid"onQueryRequest, and a route-name assertion that fails if a bypass endpoint is ever added.test_the_http_transport_exposes_no_bypass_controlis the one that will still be doing work in a year. - Doc sync is complete —
SPEC.md,SKILL.md,CHANGELOG.md,security-model.mdxandshell-tools-mixin.mdxall carry the same claims, and the developer set listed in the spec matchesDEVELOPER_COMMANDSexactly.test_a_pipe_is_refused_for_its_binary_not_for_the_pipepinning the correct reason against the issue's wrong one is a nice touch. ensure_audit_log_handlerextracted fromPathValidator._setup_audit_loggingrather than a second handler — the audit log stays one file with one rotation policy.
The bypass-control guard read `.path` off every entry in the app's route list, which raises on the `_IncludedRouter` wrapper that `include_router` leaves on the CI FastAPI version. Skipping entries without a `.path` would have walked straight past the mounted API and left the assertion passing on an empty set, so the walk now descends into sub-routers and asserts it reached a known route before checking for bypass paths.
kovtcharov-amd
commented
Sep 5, 2026
Triage on the two red checks.
Test Gaia Agent — fixed and pushed. The bypass-control guard read .path off every entry in the app's route list, which raises on the router wrapper that the CI FastAPI version leaves behind. The walk now descends into sub-routers, and it asserts it actually reached a known route first — skipping entries without a .path would have walked past the mounted API and left the assertion passing on an empty set.
The eval gate is not this PR. It is failing repo-wide right now, and the failure has nothing to do with model quality: all five scenarios error with exit code 1 after three seconds each, which is a harness failure, not a bad answer. The scorecard renders that as pass rate 75% → 0%, which reads alarming and means nothing.
The clincher is that #3372 — a CI-workflow-YAML-only change that touches no prompt, tool or model code — produces a byte-identical failure signature. #3390 is red the same way.
This should not block review of this PR, but it does block the milestone. Until #3375 lands, a failing scenario's stdout is discarded, so nobody can see why an eval fails — which is exactly why this took a log dig to characterise rather than a glance. Worth prioritising: several issues in milestone 66 have "run an eval and compare to baseline" as an acceptance criterion, and that criterion is currently unmeetable.
🟡 The newline-separator gap from the first review is still open. ls\nrm -rf /tmp/x passes validation under bypass and then has rm executed by the shell — the rm tripwire never fires and the audit record shows one phantom invocation instead of two real ones.
The new _tokenize uses whitespace_split = True, which leaves \n as whitespace rather than emitting it as a separator token. _split_pipeline therefore never sees a break, hands ['ls', 'rm', '-rf', '/tmp/x'] to _validate_bypass_command as a single segment, and ls passes. The executor runs the original string with shell=True, so the shell sees and executes both lines. The test suite added by this push covers &&, ||, ;, |, and > separators but has no newline case, so CI stays green.
🔍 Technical details
Reproducer (matches _tokenize exactly):
import shlex lexer = shlex.shlex("ls\nrm -rf /tmp/x", posix=True, punctuation_chars=";&|<>") lexer.whitespace_split = True lexer.commenters = "" print(list(lexer)) # ['ls', 'rm', '-rf', '/tmp/x'] — no separator token
_SEGMENT_SEPARATORS never fires → one segment → _validate_bypass_command("ls") → None (allowed) → executor runs the original string with shell=True → both commands execute.
The first review offered two options:
- Treat
\nas;before lexing (with a quoted-string caveat). - Refuse any multi-line command under bypass with a clear error.
Option 2 is the safe-fail path and avoids touching the lexer — add a check before _tokenize in _validate_shell_command:
if bypass and "\n" in command.strip(): return ( { "status": "error", "error": "Multi-line commands are not accepted under bypass permissions.", "has_errors": True, }, [], )
A matching test (TestOperatorsUnderBypass or TestPerSegmentWalkSurvivesBypass) should assert that "ls\nrm -rf /tmp/x" is refused under bypass and that the audit for "ls\necho hi" records two segments, not one.
...oint The guard asserted /v1/gaia/query was reachable before checking for bypass routes, and that endpoint is not mounted in the CI environment — so the guard failed there while passing locally. Keying it on the /v1/gaia/ prefix still proves the walk descended into the included router, without depending on which endpoints a given environment mounts.
kovtcharov-amd
commented
Sep 5, 2026
Correction to my triage above: the eval breakage is already tracked as #3341 — "Eval gate fails identically on main: tool_selection scenarios die in 3s and claude's stderr is swallowed" — which reaches the same conclusion independently and from the main branch rather than from a PR.
So the red eval check here needs no action on this PR. #3375 fixes the visibility half (a failing scenario's output is currently discarded), and #3368 covers the same swallowing on the judge path.
I have added #3341 to milestone 66, because several issues there carry "run an eval and compare to the committed baseline" as an acceptance criterion and that is unmeetable until it is fixed.
Bypass permissions turned off the confirmation prompt and nothing else. A user who had already granted blanket consent still could not get the agent to run a build or a test suite: compound commands like
cd build && cmake ..were refused before they parsed, and no interpreter, test runner or package manager was in the allowlist. "Verify your work before claiming it is done" was not something the agent could do, however well it was prompted. Now--bypass-permissions(and the TUI's/bypass) takes the shell's own gates off with the prompt — operators run, a developer binary set replaces the read-only policy, and the rate limit lifts.Nothing changes with bypass off.
ALLOWED_COMMANDSis untouched and the read-only tier keeps claiming exactly what it claims, so #2768's hardening of it stays meaningful. No new flag: this extends the switch that already ships.Three decisions the issues left open or got wrong, called out for review:
rmstays out;cpandmvare in. Not a boundary — anything in the set can delete a file — just a tripwire against an accidental recursive delete. Cheaper to add later than to take back.python/pytest/ghare in the set for consolidation, not new reach. All three already had paths (execute_python_file,shell:execute:skill grants). Listing them gives bypass one answer to "may this binary run" instead of three; the per-skill grant path is unchanged.pytest -q | tail -20as blocked by the operator block. It is not — pipes were never blocked; it is refused later for the ungranted binary. There is a test pinning the real reason.Closes #3373. Closes #3374.
🔍 Technical details
Where the mode lives.
PermissionStatesetsbypass_permissionson the turn's output handler alongsideauto_approve_gated_tools;ShellToolsMixin.bypass_gates_active()reads it live. Two attributes on purpose — an unattended harness that only pre-approves prompts (GAIA_AUTO_APPROVE_TOOLS) must not inherit an unguarded shell. Read live rather than cached because bypass is toggleable mid-session over the control channel; eachrun_shell_commandreads it once and threads that value through its own pre-flight and execution, so a toggle landing mid-call cannot split the two.Stdio only. The HTTP transport is a bound socket, so the handler pins
bypass_permissions = FalseandQueryRequest(extra="forbid") cannot ask for it. Narrowed from the issue's "refuse on any bound socket", which would have forbidden the sidecar's own designed configuration.Tokenisation. Bypass uses
shlex.shlex(punctuation_chars=";&|<>")so&&becomes its own token for_split_pipelineto break on; the default path still usesshlex.splitunchanged. Parens stay out of the punctuation set so operands like-name "(draft)*"survive.commentersis cleared to matchshlex.split.Audit.
gaia.security.audit_shell_commandappends to the existingfile_audit.log, called only on the bypass path so the default path is untouched. Command substitution is the one thing it cannot audit precisely — a$(...)body is not a segment.Test plan
python -m pytest tests/unit/test_shell_guardrails.py -q— 164 pass. Every bypass case asserts both states, so a binary leaking into the default tier fails here.python -m pytest hub/agents/gaia/python/tests/test_stdio.py hub/agents/gaia/python/tests/test_server_query.py -q— bypass reaches the shell gates through the realPermissionState; the HTTP transport cannot be put in bypass.test_the_parser_defaults_to_local_and_promptingis unmodified and green.printf 'Run this shell command: cd . && echo HI\n' | python -m gaia_agent.stdio→ refused with the existing operator message.--bypass-permissions→ runs,return_code: 0.tail -2 ~/.gaia/cache/file_audit.logafter the run showsSHELL | bypass | ... | segments=[["cd", "."], ["echo", "BYPASS_WORKS"]].cd tui && go build ./... && go test ./internal/ui/chat/... ./internal/cli/...tests/unit/diffed against a cleanmainworktree: identical failure and error sets (658/488 — all pre-existing on this box), +93 passing.