Skip to content

Navigation Menu

Sign in
Sign up

feat(shell): --bypass-permissions lifts the shell guardrails too - #3394

Open
kovtcharov-amd wants to merge 4 commits into
main from
feat/bypass-permissions-shell-gates
Open

feat(shell): --bypass-permissions lifts the shell guardrails too #3394
kovtcharov-amd wants to merge 4 commits into
main from
feat/bypass-permissions-shell-gates

Conversation

@kovtcharov-amd

@kovtcharov-amd kovtcharov-amd commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

Bypass permissions turned off the confirmation prompt and nothing else. A user who had already granted blanket consent still could not get the agent to run a build or a test suite: compound commands like cd build && cmake .. were refused before they parsed, and no interpreter, test runner or package manager was in the allowlist. "Verify your work before claiming it is done" was not something the agent could do, however well it was prompted. Now --bypass-permissions (and the TUI's /bypass) takes the shell's own gates off with the prompt — operators run, a developer binary set replaces the read-only policy, and the rate limit lifts.

Nothing changes with bypass off. ALLOWED_COMMANDS is untouched and the read-only tier keeps claiming exactly what it claims, so #2768's hardening of it stays meaningful. No new flag: this extends the switch that already ships.

Three decisions the issues left open or got wrong, called out for review:

  • rm stays out; cp and mv are in. Not a boundary — anything in the set can delete a file — just a tripwire against an accidental recursive delete. Cheaper to add later than to take back.
  • python/pytest/gh are in the set for consolidation, not new reach. All three already had paths (execute_python_file, shell:execute: skill grants). Listing them gives bypass one answer to "may this binary run" instead of three; the per-skill grant path is unchanged.
  • feat(shell): let --bypass-permissions lift the shell operator block #3373 cites pytest -q | tail -20 as blocked by the operator block. It is not — pipes were never blocked; it is refused later for the ungranted binary. There is a test pinning the real reason.

Closes #3373. Closes #3374.

🔍 Technical details

Where the mode lives. PermissionState sets bypass_permissions on the turn's output handler alongside auto_approve_gated_tools; ShellToolsMixin.bypass_gates_active() reads it live. Two attributes on purpose — an unattended harness that only pre-approves prompts (GAIA_AUTO_APPROVE_TOOLS) must not inherit an unguarded shell. Read live rather than cached because bypass is toggleable mid-session over the control channel; each run_shell_command reads it once and threads that value through its own pre-flight and execution, so a toggle landing mid-call cannot split the two.

Stdio only. The HTTP transport is a bound socket, so the handler pins bypass_permissions = False and QueryRequest (extra="forbid") cannot ask for it. Narrowed from the issue's "refuse on any bound socket", which would have forbidden the sidecar's own designed configuration.

Tokenisation. Bypass uses shlex.shlex(punctuation_chars=";&|<>") so && becomes its own token for _split_pipeline to break on; the default path still uses shlex.split unchanged. Parens stay out of the punctuation set so operands like -name "(draft)*" survive. commenters is cleared to match shlex.split.

Audit. gaia.security.audit_shell_command appends to the existing file_audit.log, called only on the bypass path so the default path is untouched. Command substitution is the one thing it cannot audit precisely — a $(...) body is not a segment.

Test plan

  • python -m pytest tests/unit/test_shell_guardrails.py -q — 164 pass. Every bypass case asserts both states, so a binary leaking into the default tier fails here.
  • python -m pytest hub/agents/gaia/python/tests/test_stdio.py hub/agents/gaia/python/tests/test_server_query.py -q — bypass reaches the shell gates through the real PermissionState; the HTTP transport cannot be put in bypass. test_the_parser_defaults_to_local_and_prompting is unmodified and green.
  • Refusal, real CLI: printf 'Run this shell command: cd . && echo HI\n' | python -m gaia_agent.stdio → refused with the existing operator message.
  • Execution, real CLI: same command with --bypass-permissions → runs, return_code: 0.
  • tail -2 ~/.gaia/cache/file_audit.log after the run shows SHELL | bypass | ... | segments=[["cd", "."], ["echo", "BYPASS_WORKS"]].
  • cd tui && go build ./... && go test ./internal/ui/chat/... ./internal/cli/...
  • Full tests/unit/ diffed against a clean main worktree: identical failure and error sets (658/488 — all pre-existing on this box), +93 passing.

Ovtcharov added 2 commits September 2, 2026 09:33
Running the unit suite could pop a Google OAuth consent screen on the
developer's desktop, mid-run, with a dummy client id — so the window that
stole focus was also an "Access blocked: invalid_client" error.
connectors.flow.start_authorization launches the browser from a
fire-and-forget asyncio.ensure_future task that resolves webbrowser.open
when it runs, which can be after the test that patched it has finished and
monkeypatch has restored the real function. Every connector test does patch
the launcher; the patch just isn't guaranteed to still be in place at launch
time, which is why the popup was intermittent.
Blocking the launchers for the whole session closes the race from the other
side: a per-test patch now restores to the stub, never to the real function.
Bypass permissions turned off the confirmation prompt and nothing else, so a
user who had granted blanket consent still could not get the agent to run a
build or a test suite. Compound commands were refused before they parsed, and
no interpreter, test runner or package manager was in the allowlist — "verify
your work before claiming it is done" was not something the agent could do,
however well it was prompted.
Under bypass, the shell's own gates now come off with the prompt: operators
(&&, ||, ;, >) parse and run, the read-only binary policy is replaced by a
developer set, and the rate limit is lifted. Off by default and byte-identical
to before when off — ALLOWED_COMMANDS is untouched and the read-only tier keeps
claiming exactly what it claims, so #2768's hardening of it stays meaningful.
The developer set adds node, npm, make, cmake, go, cargo, sed, awk, curl,
sleep, timeout, export, cp and mv. It also names python, python3, pytest and
gh, which are consolidation rather than new reach: those already had paths via
execute_python_file and shell:execute skill grants, and listing them here gives
bypass one answer to "may this binary run" instead of three. rm stays out — not
a boundary, since anything in the set can delete a file, but a tripwire against
an accidental recursive delete.
Stdio transport only. The HTTP surface is a bound socket, and an unguarded
shell reachable over it is remote code execution rather than a relaxed
permission model, so the handler pins bypass off and the request model forbids
unknown fields. bypass_permissions is also a separate handler attribute from
auto_approve_gated_tools: an unattended harness that merely pre-approves
prompts must not inherit an unguarded shell.
Skipping the prompt does not skip the record. Every command run under bypass is
written to file_audit.log with its full arguments and its per-segment
breakdown, so `cd build && make` is two auditable invocations rather than one
opaque string. The per-segment walk is what produces that record, so it stays
intact in both modes.
Closes #3373
Closes #3374 
@github-actions github-actions Bot added documentation Documentation changes tests Test changes security Security-sensitive changes agents tui Go terminal UI (gaia-tui) labels Sep 5, 2026

github-actions Bot commented Sep 5, 2026
edited
Loading

Copy link
Copy Markdown
Contributor

Skill audit

Skill Verdict Claimed tier Cleared tiers Findings Rules
hub/agents/gaia/npm ALLOW experimental experimental, community none

✅ All audited skills cleared the tier they claim.

Per-finding detail is withheld here on purpose. Read it in the Security > Code scanning tab, or download the skill-audit-reports artifact from this run. Offending source text is withheld from CI everywhere — reproduce it locally with gaia skill audit <dir> --show-snippets.

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Request changes

This extends --bypass-permissions so it also lifts the shell tool's own guardrails — compound commands parse, a developer binary set replaces the read-only allowlist, and the rate limit comes off — with the default path left alone. The design, the stdio-only reasoning and the test coverage are all strong. One gap is worth fixing before merge.

A command containing a line break slips past the segment walk. Under bypass, the tokenizer treats a newline as plain whitespace, so ls followed by a newline and rm -rf /tmp/x is checked as if it were a single ls invocation — it passes, and then the real shell runs both halves. Two things the PR promises quietly stop holding: the deliberate rm tripwire never fires, and the audit trail records one nonsense invocation instead of the two commands that actually ran. That matters most for the audit, because the shipped security doc tells operators that command substitution is the only thing bypass cannot audit precisely. Either treat a line break as a command separator, or refuse a multi-line command under bypass and say so.

Everything else is minor: the helper's own docstring names a gate that was never wired up, output redirection is silently swallowed for one narrow skill-granted case, and the PR quietly carries a second, unmentioned change (the test-suite browser guard).

No security escalation. The gap above sits entirely inside a mode the user deliberately turned on to get arbitrary code execution, and the default posture is unchanged — this is audit-integrity and a broken accident-tripwire, not an escalation path.

Real-world evidence

⚠️ The evidence harness failed to runevidence-bundle.md reports the stage errored before producing anything (auth / install / rate-limit; see the run logs). The changed surface was not exercised here, and this verdict rests on static review alone. I could not run the test suite either (no pytest in this environment), so the "164 pass" claim is unverified from my side. The one thing I did execute is the tokenizer, reproduced standalone, which is what confirms the newline finding above.

The PR description's test plan names exactly the right runs — the refused-then-executed cd . && echo HI pair through the real stdio CLI, and the file_audit.log line — but every box is unchecked and no captured output is pasted in. Paste those two outputs into the description and the CLI surface is covered; the TUI banner//bypass wording still wants a look on the strix-halo lane.

🔍 Technical details

Issues

🟡 A newline is not a segment separator, so a multi-line command is validated as one command and executed as several (src/gaia/agents/tools/shell_tools.py:509, :512)

_tokenize sets whitespace_split = True and leaves \n in shlex's default whitespace, so it is dropped rather than emitted as a token, and _SEGMENT_SEPARATORS never sees it. Reproduced standalone:

'ls\nrm -rf /tmp/x' -> ['ls', 'rm', '-rf', '/tmp/x'] -> segments: ['ls']
'ls; rm -rf x' -> ['ls', ';', 'rm', '-rf', 'x'] -> segments: ['ls', 'rm'] # correct

_validate_bypass_command then only ever sees ls, returns None, and run_shell_command executes with shell=True on the original string (exec_cmd = command), so the shell runs both. Consequences:

  1. rm is reachable under bypass without the refusal the set exists to produce. Explicitly not a security boundary per your own note — but it is the accident tripwire, and this is exactly the accident it was for.
  2. _audit_shell_execution writes segments=[["ls","rm","-rf","/tmp/x"]] — one invocation that never happened, instead of the two that did. docs/plans/security-model.mdx ("Command substitution is the one thing bypass cannot audit precisely") and SPEC.md both become inaccurate.

Rewriting \n to ; before lexing would fix both but mis-splits a literal newline inside a quoted operand. The safe-fail version, if you'd rather not touch tokenisation:

 bypass = self.bypass_gates_active()
 if bypass and "\n" in command.strip():
 return (
 {
 "status": "error",
 "error": (
 "Multi-line commands are not accepted, even with bypass "
 "permissions active."
 ),
 "has_errors": True,
 "hint": (
 "A line break is a command separator the per-segment "
 "audit record cannot see. Join the commands with '&&' "
 "or ';', or put the script in a file and run it."
 ),
 },
 [],
 )

(goes in _validate_shell_command right after bypass = self.bypass_gates_active(), before the operator check.) Whichever route you pick, add a case to TestPerSegmentWalkSurvivesBypasscheck("ls\nrm -rf /tmp/foo", bypass=True) is not None — since that class is precisely where a reader will look for this guarantee.

🟢 bypass_gates_active's docstring names a gate that doesn't read it (shell_tools.py:582)

Every gate reads this — _validate_shell_command, skill_grant_covers_call, the tool description, the executor

The run_shell_command description is a static string (shell_tools.py:910) that still says "Pipes (|) are supported" and gives only read-only examples. So the model is never told the surface widened, which is the PR's stated goal ("verify your work before claiming it is done"). Either drop the clause from the docstring, or make the description bypass-aware — the latter is a tool-description change and would need an eval run per CLAUDE.md, so the docstring edit is the cheap correct fix if you want to keep the scope tight.

🟢 Redirection is silently swallowed for a lone skill-granted segment under bypass (shell_tools.py:1128)

> is not in _SEGMENT_SEPARATORS, so gh issue list > out.txt is one segment; with a shell:execute:gh grant active lone_granted_segment is true, use_shell stays false, and gh receives literal > and out.txt as argv. Narrow (needs both a grant and bypass), and keeping argv for granted binaries is deliberate and right — but the result is a wrong answer rather than a refusal. Worth either excluding redirect tokens from lone_granted_segment, or a comment saying redirection is intentionally inert on that path.

🟢 An unmentioned second thread rides along (tests/conftest.py:38, tests/unit/test_no_real_browser_launch.py)

The session-scoped browser guard is a good fix and well-explained in its own docstring, but it is unrelated to bypass permissions and gets no line in the PR description — a reviewer has to discover it from the diff. Worth a sentence in the description, or its own PR. Minor gap while you're there: it covers webbrowser.open/open_new/open_new_tab but not webbrowser.get(...).open(...), which is the other way start_authorization-style code reaches a real browser.

Strengths

  • Every bypass test asserts both states. TestDeveloperBinariesRefusedByDefault and TestReadOnlySubGuardsLiftUnderBypassOnly pin the default tier alongside the bypass tier, so a binary leaking into ALLOWED_COMMANDS fails here rather than shipping. test_allowed_commands_carries_no_developer_binary is the right invariant to have written down.
  • The tests go past validation into real execution. TestExecutorUnderBypass spawns actual processes, which is what catches the two things validation-only tests cannot: that the operators reach a shell at all, and the lazily-created rate-limit deque that bypass skips creating. test_the_rate_limit_is_lifted's docstring names the AttributeError it exists to prevent.
  • The two-attribute split is the right call and is enforced. bypass_permissions separate from auto_approve_gated_tools, with test_auto_approve_alone_does_not_lift_the_shell_gates pinning it, means GAIA_AUTO_APPROVE_TOOLS can never quietly inherit an unguarded shell.
  • The HTTP transport is pinned shut from three directions — the handler assignment, extra="forbid" on QueryRequest, and a route-name assertion that fails if a bypass endpoint is ever added. test_the_http_transport_exposes_no_bypass_control is the one that will still be doing work in a year.
  • Doc sync is completeSPEC.md, SKILL.md, CHANGELOG.md, security-model.mdx and shell-tools-mixin.mdx all carry the same claims, and the developer set listed in the spec matches DEVELOPER_COMMANDS exactly. test_a_pipe_is_refused_for_its_binary_not_for_the_pipe pinning the correct reason against the issue's wrong one is a nice touch.
  • ensure_audit_log_handler extracted from PathValidator._setup_audit_logging rather than a second handler — the audit log stays one file with one rotation policy.

The bypass-control guard read `.path` off every entry in the app's route
list, which raises on the `_IncludedRouter` wrapper that `include_router`
leaves on the CI FastAPI version. Skipping entries without a `.path` would
have walked straight past the mounted API and left the assertion passing on
an empty set, so the walk now descends into sub-routers and asserts it
reached a known route before checking for bypass paths.

Copy link
Copy Markdown
Collaborator Author

Triage on the two red checks.

Test Gaia Agent — fixed and pushed. The bypass-control guard read .path off every entry in the app's route list, which raises on the router wrapper that the CI FastAPI version leaves behind. The walk now descends into sub-routers, and it asserts it actually reached a known route first — skipping entries without a .path would have walked past the mounted API and left the assertion passing on an empty set.

The eval gate is not this PR. It is failing repo-wide right now, and the failure has nothing to do with model quality: all five scenarios error with exit code 1 after three seconds each, which is a harness failure, not a bad answer. The scorecard renders that as pass rate 75% → 0%, which reads alarming and means nothing.

The clincher is that #3372 — a CI-workflow-YAML-only change that touches no prompt, tool or model code — produces a byte-identical failure signature. #3390 is red the same way.

This should not block review of this PR, but it does block the milestone. Until #3375 lands, a failing scenario's stdout is discarded, so nobody can see why an eval fails — which is exactly why this took a log dig to characterise rather than a glance. Worth prioritising: several issues in milestone 66 have "run an eval and compare to baseline" as an acceptance criterion, and that criterion is currently unmeetable.

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

🟡 The newline-separator gap from the first review is still open. ls\nrm -rf /tmp/x passes validation under bypass and then has rm executed by the shell — the rm tripwire never fires and the audit record shows one phantom invocation instead of two real ones.

The new _tokenize uses whitespace_split = True, which leaves \n as whitespace rather than emitting it as a separator token. _split_pipeline therefore never sees a break, hands ['ls', 'rm', '-rf', '/tmp/x'] to _validate_bypass_command as a single segment, and ls passes. The executor runs the original string with shell=True, so the shell sees and executes both lines. The test suite added by this push covers &&, ||, ;, |, and > separators but has no newline case, so CI stays green.

🔍 Technical details

Reproducer (matches _tokenize exactly):

import shlex
lexer = shlex.shlex("ls\nrm -rf /tmp/x", posix=True, punctuation_chars=";&|<>")
lexer.whitespace_split = True
lexer.commenters = ""
print(list(lexer)) # ['ls', 'rm', '-rf', '/tmp/x'] — no separator token

_SEGMENT_SEPARATORS never fires → one segment → _validate_bypass_command("ls") → None (allowed) → executor runs the original string with shell=True → both commands execute.

The first review offered two options:

  1. Treat \n as ; before lexing (with a quoted-string caveat).
  2. Refuse any multi-line command under bypass with a clear error.

Option 2 is the safe-fail path and avoids touching the lexer — add a check before _tokenize in _validate_shell_command:

if bypass and "\n" in command.strip():
 return (
 {
 "status": "error",
 "error": "Multi-line commands are not accepted under bypass permissions.",
 "has_errors": True,
 },
 [],
 )

A matching test (TestOperatorsUnderBypass or TestPerSegmentWalkSurvivesBypass) should assert that "ls\nrm -rf /tmp/x" is refused under bypass and that the audit for "ls\necho hi" records two segments, not one.

...oint
The guard asserted /v1/gaia/query was reachable before checking for bypass
routes, and that endpoint is not mounted in the CI environment — so the
guard failed there while passing locally. Keying it on the /v1/gaia/ prefix
still proves the walk descended into the included router, without depending
on which endpoints a given environment mounts.

Copy link
Copy Markdown
Collaborator Author

Correction to my triage above: the eval breakage is already tracked as #3341"Eval gate fails identically on main: tool_selection scenarios die in 3s and claude's stderr is swallowed" — which reaches the same conclusion independently and from the main branch rather than from a PR.

So the red eval check here needs no action on this PR. #3375 fixes the visibility half (a failing scenario's output is currently discarded), and #3368 covers the same swallowing on the judge path.

I have added #3341 to milestone 66, because several issues there carry "run an eval and compare to the committed baseline" as an acceptance criterion and that is unmeetable until it is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agents documentation Documentation changes security Security-sensitive changes tests Test changes tui Go terminal UI (gaia-tui)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(shell): one developer binary policy, not three partial paths feat(shell): let --bypass-permissions lift the shell operator block

1 participant

AltStyle によって変換されたページ (->オリジナル) /