Skip to content

Navigation Menu

Sign in
Sign up

say which invariants a run could not satisfy when it gives up - #22

Open
RyanRana wants to merge 1 commit into
boldsoftware:main from
RyanRana:pr-plan-rejection-diagnostics
Open

say which invariants a run could not satisfy when it gives up #22
RyanRana wants to merge 1 commit into
boldsoftware:main from
RyanRana:pr-plan-rejection-diagnostics

Conversation

@RyanRana

@RyanRana RyanRana commented Aug 7, 2026

Copy link
Copy Markdown

a run that exhausts its turns is the expensive failure: it spends the whole token budget and then reports only that it gave up.

meat: agent did not submit within 24 turns

that is the entire output. there is no way to tell whether the model was close, which validator it kept hitting, or whether the diff was simply too hard.

this records the validator errors behind every rejected preview_plan and submit, buckets them by invariant with coordinates stripped so the same failure at different lines collapses to one line, and reports the worst offenders when the loop exits without a submission:

meat: agent did not submit within 4 turns (in=400 out=80 tokens); invariants it could not satisfy:
 4x (turns 1,2,3,4) replace[]: line N is outside the diff (N-N)

observability only. no model-visible strings change, so abridgeProtocolVersion and the pinned rubric hash are untouched, and no cached result is invalidated.

context for why this is worth having: abridging a 23.7KB commit with claude-sonnet-4.5 spent all 24 turns, cost 1ドル.75, and produced nothing. the compiler has 53 rejection sites across 12 validators and plans are complete restatements, so a single bad coordinate discards the whole plan. which invariant was actually blocking that run is currently unknowable.

a run that exhausts its turns is the expensive failure: it spends the whole
token budget and then reports only that it gave up. there is no way to tell
whether the model was close, or which validator it kept hitting.
record the validator errors behind every rejected preview_plan and submit,
bucketed by invariant with coordinates stripped, and report the worst
offenders when the loop exits without a submission.
observability only. no model-visible strings change, so the protocol
version and rubric hash are untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

AltStyle によって変換されたページ (->オリジナル) /