CI gates for AI-generated PRs need re-derivable evidence

DEV Community

Building deterministic developer tools for AI-assisted engineering. Creator of Agent Gate for AI PRs.

Joined

Jun 11, 2026

Thanks — this is a really good distinction.

I agree that a stable findingId plus an evidenceSnapshot does not magically make the whole runner state reproducible.

The way I’m thinking about it is:

findingId is just the audit handle
evidenceSnapshot is the deterministic preimage that the rule actually consumed
it is not a full claim that the entire CI world state can be recreated later

So for the current GHA rules, the snapshot is mostly trying to answer:

"Given the rule input Agent Gate saw, can someone re-derive why this finding fired?"

not:

"Can someone recreate the exact runner environment that produced the whole CI run?"

That boundary matters.

For the portability question: I’d like the evidence model to stay portable, but individual rules can absolutely be platform-specific.

workflow/permission-escalation is GHA-specific by design. Outside GitHub Actions, I’d expect the evidence preimage to look different — maybe job identity, runner image, pipeline config diffs, permission/capability changes, or whatever that CI system exposes as stable structured input.

Your point about fresh ephemeral VMs and structured test reports is especially useful for the "missing test evidence" side. Today Agent Gate’s missing-test check is intentionally conservative: it is change/self-consistency evidence, not proof of behavioral coverage. Stable test IDs from structured reports would make that rule much stronger than trying to infer too much from logs.

So I think the clearer framing is:

Agent Gate evidence should be re-derivable from the material the rule records
runner/platform reproducibility is a separate layer
better CI substrates can make the recorded evidence much stronger
some rules are portable in shape, but not in their concrete evidence source

This is probably worth spelling out in the evidence model docs. Appreciate the Semaphore perspective.