-
Notifications
You must be signed in to change notification settings - Fork 0
Pipeline Design 444
Now I have a complete picture of all three root causes. Here is the ADR:
Design: fix(harness): goal/original_goal in loop-state.md accumulate synthesized content — unbounded prompt inflation
The Shipwright harness stores pipeline state in .claude/loop-state.md (frontmatter YAML). Two fields track the task objective:
-
goal:— mutable; receives appended synthesis (test failures, review notes, human feedback) each iteration -
original_goal:— intended to be immutable; the unchanged user-supplied task description
The harness uses original_goal to anchor the ## Your Goal section in Claude's prompt during stuckness recovery, and as the baseline when resetting after a restart. When both fields accumulate synthesized content, every subsequent iteration's prompt contains a larger "Your Goal" block — unbounded prompt inflation.
Three independent root causes combine to corrupt original_goal:
RC1 — Lazy bootstrap in write_state() (pipeline-state.sh:610-615)
local _write_goal="${ORIGINAL_GOAL:-$GOAL}" if [[ -z "${ORIGINAL_GOAL:-}" && -n "${_write_goal}" ]]; then ORIGINAL_GOAL="$_write_goal" fi
In --issue pipeline runs, GOAL is populated by the intake stage after pipeline_start() has already called write_state(). The first call to write_state() with a non-empty GOAL bootstraps ORIGINAL_GOAL from whatever $GOAL contains at that moment — which may already include synthesized content appended by intake processing.
RC2 — Identical escaping for both fields (pipeline-state.sh:616-622)
local _goal_esc="${_write_goal//\\/\\\\}" # _write_goal = ORIGINAL_GOAL:-$GOAL ... printf 'goal: "%s"\n' "$_goal_esc" printf 'original_goal: "%s"\n' "$_goal_esc"
Both fields are written using the same _goal_esc derived from the same source variable. original_goal: is never written from a separate, protected source. If ORIGINAL_GOAL is contaminated (RC1), both fields permanently store contaminated content on every subsequent write_state() call.
RC3 — compose_prompt() uses $GOAL in the non-stuck branch (loop-iteration.sh:369)
local prompt_goal="$GOAL" # mutable, accumulated if [[ "$stuckness_detected" == "true" ]]; then local _base_goal="${ORIGINAL_GOAL:-$GOAL}" # only stuck path uses ORIGINAL_GOAL ... fi
In normal (non-stuck) iterations, prompt_goal is the full accumulated $GOAL, so even a correct ORIGINAL_GOAL provides no relief — the prompt still grows.
Constraint: All scripts must be Bash 3.2 compatible (set -euo pipefail, no associative arrays, no ${var,,}).
Three surgical changes, no new state fields, no new files.
Delete lines 613-615 entirely:
# DELETE THIS BLOCK: if [[ -z "${ORIGINAL_GOAL:-}" && -n "${_write_goal}" ]]; then ORIGINAL_GOAL="$_write_goal" fi
ORIGINAL_GOAL must be set before write_state() is first called. The lazy bootstrap is the contamination vector — it silently promotes a possibly-mutated $GOAL into the immutable slot. The two legitimate callers that need this are sw-loop.sh:419 (already correct) and the --issue pipeline path (fixed by Change 2).
Write the two fields from independent sources:
local _goal_esc="${GOAL//\\/\\\\}" _goal_esc="${_goal_esc//$'\n'/\\n}" local _orig_esc="${ORIGINAL_GOAL//\\/\\\\}" _orig_esc="${_orig_esc//$'\n'/\\n}" ... printf 'goal: "%s"\n' "$_goal_esc" printf 'original_goal: "%s"\n' "$_orig_esc"
This makes original_goal: a true snapshot of $ORIGINAL_GOAL in memory. If ORIGINAL_GOAL is empty (e.g., a caller forgot to set it), the written value is an empty string — visible and debuggable rather than silently contaminated.
Change the normal-path assignment:
# Before: local prompt_goal="$GOAL" # After: local prompt_goal="${ORIGINAL_GOAL:-$GOAL}"
The stuckness branch already truncates _base_goal to the first paragraph; the normal branch can use the full ORIGINAL_GOAL since it is the clean user-supplied text. The :-$GOAL fallback preserves backward compatibility when ORIGINAL_GOAL is empty (legacy state files without the field).
The intake stage appends build context to $GOAL and is the primary source of contamination. The pipeline startup (pipeline_start() in pipeline-state.sh) must set ORIGINAL_GOAL before calling write_state(). The call sequence is:
pipeline_start() → sets PIPELINE_NAME, GOAL (from --goal arg or empty) → write_state()
intake stage → appends to GOAL → write_state()
At the pipeline_start() call, GOAL is the clean user-supplied string. Adding ORIGINAL_GOAL="${ORIGINAL_GOAL:-$GOAL}" immediately before the first write_state() call inside pipeline_start() (or at the top of each pipeline-stage entry point when ORIGINAL_GOAL is unset) ensures the immutable baseline is captured from the clean value.
-
Store
original_goalonly once (write-once sentinel) — Guardwrite_state()sooriginal_goal:is only written on the first call, never overwritten.- Pros: Simple, zero drift.
- Cons: Complicates
write_state()with file-read-before-write; breaks atomicity guarantee; fails when state file is deleted for a fresh start.
-
Strip synthesized content from
$GOALbefore writingoriginal_goal:— Apply the existing sentinel-stripping logic (legacy backward-compat block inresume_state(), lines 720-728) insidewrite_state()on every call.- Pros: Self-healing — eventually converges even from a contaminated state.
- Cons: Sentinel patterns must be kept in sync across two sites; stripping is fragile (regex-free bash string matching can miss variants); does not address RC3 (prompt still uses
$GOAL).
-
Make
$GOALimmutable; use a separate$ENRICHED_GOALfor synthesis — Full rename of the mutable accumulator.- Pros: Clean separation at the variable level.
- Cons: High blast radius —
$GOALis referenced in >50 locations across 8 scripts; invasive rename increases merge-conflict risk and regression surface.
-
Persist
original_goalas a separate file — Write$ORIGINAL_GOALto.claude/original-goal.txtonce at pipeline start; read it back on resume.- Pros: Completely isolated from state-file corruption.
- Cons: Adds a new artifact to track, clean up, and document; splits state across files when the state-file format already supports the field.
| File | Line(s) | Change |
|---|---|---|
scripts/lib/pipeline-state.sh |
610-617 | Remove lazy bootstrap block; split _goal_esc / _orig_esc variables |
scripts/lib/pipeline-state.sh |
621-622 | Write goal: from _goal_esc, original_goal: from _orig_esc
|
scripts/lib/pipeline-state.sh |
~185-200 | Add ORIGINAL_GOAL="${ORIGINAL_GOAL:-$GOAL}" before first write_state() inside pipeline_start()
|
scripts/lib/loop-iteration.sh |
369 | Change local prompt_goal="$GOAL" → local prompt_goal="${ORIGINAL_GOAL:-$GOAL}"
|
None.
None — no new packages or external dependencies.
-
pipeline_start()call ordering: The newORIGINAL_GOAL="${ORIGINAL_GOAL:-$GOAL}"line inpipeline_start()must execute before the firstwrite_state()call within that function. Readpipeline-state.sh:185-201carefully; there are two earlywrite_state()calls (intake gate and initial-state write). -
Resume path (
resume_state()) backward compatibility: The_has_original_goal=falsepath (lines 717-730) correctly handles legacy state files with nooriginal_goal:field. After this fix,original_goal:will always be written, so the legacy path becomes dead code — leave it for now; removing it is a separate cleanup. -
Empty
ORIGINAL_GOALatcompose_prompt()call time: The:-$GOALfallback in Change 3 handles this. Test that the fallback does not re-introduce inflation by verifying$GOALis the uncontaminated value whenORIGINAL_GOALis genuinely absent (loop mode, first iteration before any state file exists). -
Concurrent writers:
write_state()already usesmv -f tmp → STATE_FILEatomically. The split-variable change does not break atomicity.
┌─────────────────────────────────────────────────────────────────┐
│ pipeline_start() / sw-loop.sh:419 │
│ Sets ORIGINAL_GOAL="${ORIGINAL_GOAL:-$GOAL}" (immutable seed) │
└───────────────────────────┬─────────────────────────────────────┘
│ ORIGINAL_GOAL (read-only after this point)
┌─────────────────┼───────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌─────────────┐ ┌───────────────────────┐
│ write_state() │ │ resume_ │ │ compose_prompt() │
│ pipeline- │ │ state() │ │ loop-iteration.sh │
│ state.sh │ │ pipeline- │ │ │
│ │ │ state.sh │ │ prompt_goal = │
│ goal: ← $GOAL │ │ │ │ ORIGINAL_GOAL:-$GOAL │
│ original_goal: │ │ restores │ │ (not $GOAL) │
│ ← $ORIGINAL_ │ │ both vars │ └───────────────────────┘
│ GOAL (fixed) │ │ from file │
└──────────────────┘ └─────────────┘
│
▼
.claude/loop-state.md
goal: "<mutable, may grow>"
original_goal: "<immutable user text>"
# write_state() — no args, reads globals # Precondition: ORIGINAL_GOAL must be set before first call (non-empty user goal) # Postcondition: loop-state.md contains two distinct fields: # goal: encodes current $GOAL (mutable) # original_goal: encodes $ORIGINAL_GOAL (must not change across calls) # Error contract: returns 1 if disk space < 100 MB; returns 0 on success write_state() # compose_prompt() — no args, reads globals, returns prompt string on stdout # Precondition: ORIGINAL_GOAL set (or empty — fallback to $GOAL via :-) # Postcondition: ## Your Goal section contains ORIGINAL_GOAL, not accumulated GOAL # Error contract: never exits; empty output triggers caller-side error compose_prompt() # pipeline_start() — call site must set ORIGINAL_GOAL before first write_state() call # Precondition: GOAL set from user input (clean, pre-synthesis) # Postcondition: ORIGINAL_GOAL == GOAL at moment of first write_state() call pipeline_start()
User supplies --goal "fix auth bug"
│
▼
GOAL="fix auth bug"
ORIGINAL_GOAL="fix auth bug" ← set once here (pipeline_start / sw-loop.sh:419)
│
▼
write_state() ──► loop-state.md:
goal: "fix auth bug"
original_goal: "fix auth bug"
│
intake appends test failure context to GOAL
│
▼
GOAL="fix auth bug\n\nBLOCKING ISSUES: ..."
ORIGINAL_GOAL="fix auth bug" ← unchanged
│
▼
write_state() ──► loop-state.md:
goal: "fix auth bug\n\nBLOCKING ISSUES: ..."
original_goal: "fix auth bug" ← protected
│
▼
compose_prompt() uses ORIGINAL_GOAL → ## Your Goal: "fix auth bug"
| Component | Errors it handles | Propagation |
|---|---|---|
write_state() |
Disk-full → returns 1; mv failure → logs and returns 1 | Caller logs; pipeline continues (non-fatal) |
compose_prompt() |
Empty ORIGINAL_GOAL → falls back to $GOAL via :-
|
Silent fallback; no exit |
pipeline_start() |
GOAL empty at call time → ORIGINAL_GOAL set to "" → original_goal: written as empty string |
Visible in state file; downstream :-$GOAL fallbacks activate |
resume_state() |
Missing original_goal: field (legacy file) → sets ORIGINAL_GOAL via sentinel-stripping compat path |
Safe; existing behavior preserved |
- After
write_state()is called twice with$GOALmutated between calls,original_goal:inloop-state.mdequals the value of$GOALat the time of the first call. -
compose_prompt()output for the## Your Goalsection matches$ORIGINAL_GOAL, not$GOAL, on both stuck and non-stuck iterations. - A pipeline started with
--issue <N>(intake path) produces a state file whereoriginal_goal:does not contain any appended synthesis content (no "BLOCKING ISSUES", "Previous build", "KNOWN FIX" prefixes). -
resume_state()round-trips a multi-lineORIGINAL_GOALwithout corruption (existing test insw-lib-pipeline-state-test.sh:486covers this; confirm it passes unchanged). -
npm testpasses with no new failures. - Manual inspection: run
cat .claude/loop-state.mdafter two iterations of a loop with a test failure injected —original_goal:must be identical in both snapshots.