AGENTS.md Is Not Enough for Safe AI Agent Execution

DEV Community

building Ota so running any repo is safe no longer guesswork.

Location

London, England
Joined

Jul 4, 2021

Yes, fair push.

Ota is stronger than AGENTS.md because execution is contract-based and receipts are harness-authored, not agent-authored. But you’re right that safe_tasks is not yet a full runner-enforced allowlist.

Today, Ota can restrict execution to declared contract tasks, and it uses safe_for_agent / agent.safe_tasks for visibility, governance, and review. What is still missing is an agent-scoped execution mode where the runner refuses declared tasks outside the safe set.

So this is a real product gap, and your framing is the right bar. We’ll prioritize it.

Curious how you enforce that today. Are you using a custom harness, a sandboxed runner, or a tool we should look at?

vinimabreu profile image

Vinicius Pereira

AI and data engineer. RAG, agents, automations, and scraping pipelines, all built with tests and CI. I write about the small reliable pieces I reuse.

Pronouns

he/him
Work

AI and data engineer, freelance
Joined

Jun 30, 2026

• Jul 1

Honestly it's less a product and more a discipline, I'm not running some slick custom sandbox. The rule I hold is that the agent never sits in the enforcement path, so two things.

First, a capability boundary instead of a preference. The agent only ever gets a fixed set of typed tools, so an undeclared action isn't deprioritized, it just isn't callable, there's no shell to reach for in the first place. Where I do use an agent runner, I lean on the harness being able to hard-deny a call before it executes rather than the model choosing to behave, e.g. Claude Code's permission modes + pre-tool hooks do exactly this, a blocked call is the runner refusing, not the model deciding. That's the closest off-the-shelf thing to what you're describing that I'd point you at.

Second, the receipt has to live one layer down from the thing being judged. For code work that's just CI as the runner, the agent proposes a diff but the gate outside it decides pass/fail and emits the evidence, and it can't merge past a red gate no matter what it thinks it did. I've got a small project (bedrock) built around this, the reliability eval runs in CI against a known-good key and fails the build, so the agent can't author its own green check, the harness does. The second the agent can write the receipt or reach past its tool set you're back to narration w/ extra steps, so the whole game is keeping the thing that verifies separate from the thing that acts. safe_tasks going runner-enforced is basically that, and imo it's the right thing to prioritize.

b0bai profile image

Bobai Kato Ota

Bobai Kato

building Ota so running any repo is safe no longer guesswork.

Location

London, England
Joined

Jul 4, 2021

• Jul 1

That makes sense, but it also shows why discipline alone feels transitional to me.

If the boundary lives in operator habit, local harness setup, or "we know not to give the agent a shell," it can work for one careful team, but it gets harder to scale cleanly across repos, teams, and orgs. What do you think?`

The thing I keep coming back to is: whose discipline is the control once this spreads? The individual operator, the team, platform engineering, or security? And how do you keep that boundary consistent when different teams use different agents and runners?

That is the part we want Ota to handle better: declare the allowed execution surface in the repo contract, let the runner enforce it, and keep the receipt and evidence outside the agent.

Curious how you think about that scaling problem today.

vinimabreu profile image

Vinicius Pereira

AI and data engineer. RAG, agents, automations, and scraping pipelines, all built with tests and CI. I write about the small reliable pieces I reuse.

Pronouns

he/him
Work

AI and data engineer, freelance
Joined

Jun 30, 2026

• Jul 1

Yeah, I'll concede the framing, discipline as the control doesn't scale, and honestly it's the same critique I'd make of my own answer. "We know not to give the agent a shell" is just a prompt guardrail wearing an ops hat. So declaring the surface in the repo and having a runner enforce it is the right direction, no argument there.

Where I'd push is on where the teeth actually live once it spreads, because I don't think a repo contract plus a runner fully closes it either. The runner is per-dev and opt-in, so the moment one team points a different agent at the repo, or just runs the thing outside Ota, the contract is back to being a suggestion for them. You've moved the boundary from operator habit to runner adoption, which is genuinely better, but it's still a boundary someone can route around by not using the runner. So to me the durable version is the contract compiling down to enforcement at the points that are already mandatory and agent-agnostic across an org: CI required checks and branch protection on the merge side, and the sandbox plus capability and egress boundary on the execution side. Those don't care which agent or runner you used, they sit at a chokepoint nobody opts out of because it's the same gate human PRs already go through.

So I'd frame Ota's contract as the spec, the single declared source of what's allowed, and then the scaling answer is less "everyone adopts the same runner" and more "that spec is enforced at the org chokepoints that already exist and are already non-optional." The local runner enforcing it is great for fast feedback, but imo that's the best-effort layer, not the teeth. Whose discipline is the control? Ideally nobody's, it's the merge gate and the sandbox, same as it already is for the humans.

b0bai profile image

Bobai Kato Ota

Bobai Kato

building Ota so running any repo is safe no longer guesswork.

Location

London, England
Joined

Jul 4, 2021

• Jul 2

That’s a strong point, and I think it pushes Ota in the right direction. I appreciate the feedback there.

I agree the local runner is not the final teeth if using it is still optional. That makes it the fast-feedback layer, not the durable control.

The stronger shape is:

ota.yaml as the declared spec
local runner enforcement for immediate feedback
CI required checks and branch protection as the merge gate
sandbox, capability, and egress controls as the execution boundary

That feels like the real scaling path, because it does not depend on everyone using the same agent or even the same runner. The contract stays the source of truth, but the enforcement lives at the chokepoints teams already cannot opt out of.

Curious where you’d start first if you were building that out: compile the contract into CI required checks, or into sandbox/capability policy for the runner?

vinimabreu profile image

Vinicius Pereira

AI and data engineer. RAG, agents, automations, and scraping pipelines, all built with tests and CI. I write about the small reliable pieces I reuse.

Pronouns

he/him
Work

AI and data engineer, freelance
Joined

Jun 30, 2026

• Jul 2

I'd start with CI required checks, mostly for the reason you already named: it's the chokepoint nobody can opt out of, and it's one you inherit for free. Branch protection is already sitting in every repo, so you get a non-optional gate without building a new enforcement surface, and it holds even when people are on different runners or agents.

But I'd be clear about what that actually buys you. CI can only enforce the declared intent, whether the change conforms to ota.yaml at merge time. It can't see what the thing does once it runs. So it's the right first gate, but a green check means "this matches the contract," not "this can't misbehave." The sandbox and capability layer is what closes that gap, because it enforces at execution, where the unpredictable behavior actually shows up.

So the order I'd go is CI first, then compile the same ota.yaml into the sandbox and capability policy, so the runtime boundary is derived from the same source of truth instead of a second policy you hand-maintain. The failure I'd want to avoid is the two drifting apart, where CI says compliant and the runner still allows something the spec quietly stopped describing. One contract, two enforcement points, same file.

b0bai profile image

Bobai Kato Ota

Bobai Kato

building Ota so running any repo is safe no longer guesswork.

Location

London, England
Joined

Jul 4, 2021

• Jul 2

That makes sense.

That’s the direction I'll be pushing Ota toward now: one contract, multiple enforcement points.

The part I’m also most interested in is the runtime side, because CI is clearer as a first enforcement point. On the sandbox/capability side, what are you actually trusting today: filesystem boundaries, tool allowlists, network/egress controls, or something else? Even a rough example from a real repo flow would be useful.

vinimabreu profile image

Vinicius Pereira

AI and data engineer. RAG, agents, automations, and scraping pipelines, all built with tests and CI. I write about the small reliable pieces I reuse.

Pronouns

he/him
Work

AI and data engineer, freelance
Joined

Jun 30, 2026

• Jul 2

honestly, of the three, network egress is the one i trust most and instrument first, because it's where the irreversible stuff lives. filesystem and tool allowlists protect the repo; egress protects the outside world (prod writes, exfil, paid-api calls), and it's the hardest to fake compliance on. so my default is deny-all egress with an allowlist of the exact hosts a flow legitimately needs.

concrete example from a data pipeline i run: it pulls from one upstream source and writes to a google sheet, an airtable base, and a slack webhook, nothing else. so the capability boundary is literally those hosts on the egress allowlist, plus the repo and config mounted read-only and a single writable outputs/cache dir. if the process reaches any other host or tries to write outside that dir, it's already outside the contract and it dies there. i don't need it to declare intent, the boundary just doesn't exist for it. that maps onto your model cleanly: ota.yaml declares the allowed hosts and the writable paths, CI checks they're declared, and the runtime compiles that same block into default-deny egress plus a read-only mount.

the honest part: i trust tool/command allowlists the least on their own, because one broad tool reopens everything. a shell node, or an llm step that can emit arbitrary code, and the neat allowlist is decorative. so i don't lean on the tool list as the boundary, i keep the tool surface narrow and let egress and filesystem be the real backstop, on the assumption that any tool might do more than it says. it's cheaper to constrain what a step can reach and touch than to enumerate everything it might try to do.

b0bai profile image

Bobai Kato Ota

Bobai Kato

building Ota so running any repo is safe no longer guesswork.

Location

London, England
Joined

Jul 4, 2021

• Jul 2

This is very useful. The key point is that the real boundary is reach and touch, not just command choice. Tool allowlists can narrow the surface, but egress and filesystem are the harder backstops because they constrain what execution can actually affect.
That maps cleanly to direction I want Ota to go: contract-declared hosts, writable paths, and runtime compilation into default-deny egress plus explicit writable boundaries from the same ota.yaml.

One thing I’m curious about from your example: is host-level allowlisting usually enough, or do you often want it narrower than that?

vinimabreu profile image

Vinicius Pereira

AI and data engineer. RAG, agents, automations, and scraping pipelines, all built with tests and CI. I write about the small reliable pieces I reuse.

Pronouns

he/him
Work

AI and data engineer, freelance
Joined

Jun 30, 2026

• Jul 2

host-level is usually where i start, but it's only actually enough when the host is single-purpose and i own the whole surface. the slack webhook is the clean case: that host only accepts a post to one webhook path, so the host basically IS the capability and nothing narrower buys me anything.

where it stops being enough is the multi-capability hosts. sheets.googleapis.com and api.airtable.com are the same host for every sheet and every base on the planet. allow the host and a compromised step can still read or write any sheet or base the token can reach, not just the one output. so on those i don't try to write a finer network rule, i narrow the credential instead: the service account is shared onto that one sheet, the airtable PAT is scoped to that one base. the egress rule says which host, the token scope says what on that host, and a wrong destination is useless even when the step reaches it.

the reason i don't push the network rule below host is that TLS hides the path, so sub-host filtering means running a proxy that can see the request, which is a lot of moving parts for what credential scoping already gives me for free. the one place that logic breaks is shared object storage. an s3 or gcs bucket sits behind a host thousands of tenants share, so host-allow there still lets you exfil to your own bucket on the same name. that's the case where i'll actually reach for a per-bucket hostname or a private endpoint, because the credential trick doesn't save me.

so the way i'd put it: host-level is the right granularity for the network layer, but it's rarely the whole boundary on its own. for shared apis the tighter boundary is a token that can only address one thing, and i'd trust that over any finer network rule because it holds even when a step reaches the host it shouldn't.

View full discussion (11 comments)