-
Notifications
You must be signed in to change notification settings - Fork 1
Pipeline Design 242
The _extract_text_from_json function in scripts/sw-loop.sh (line 555) parses Claude CLI JSON output to extract .result text for iteration logs. The function uses a first_char dispatch pattern: it reads the first byte of the output file to determine the JSON shape and route to the appropriate parser.
The bug: The original code only checked for [ (JSON array) when deciding whether to use jq. When Claude returned a JSON object ({...}) — which happens in certain Claude CLI output modes — the function skipped the jq parsing path entirely and fell through to Case 3, which emitted warn "JSON output but jq not available — using raw output". This warning was factually wrong: jq was available, the code simply didn't recognize { as valid JSON input.
Constraints:
- Bash 3.2 compatible (macOS default) — no associative arrays or advanced string ops
- The function is called on every loop iteration, so it must be fast and never crash
- jq may genuinely not be installed on some systems — the "jq not available" warning must remain for that case
- Claude CLI can return JSON in multiple shapes: arrays (
[{...}]), objects ({...}), plain text, or empty output
Extend the first_char dispatch to recognize both [ and { as valid JSON, routing both to the jq parsing path. Within that path, branch on first_char to apply the correct jq expression for each shape:
Data flow (4 cases, unchanged structure):
-
Empty file — write
(no output)placeholder -
JSON array or object (jq available) — try
.result, fall back to.content, then placeholder - JSON array or object (jq unavailable) — warn + copy raw file (now accurately reports jq absence)
- Plain text — copy raw file
Extraction logic within Case 2:
- Arrays:
.[-1].result // emptythen.[].content // empty(unchanged from original) - Objects:
.result // emptythen.content // empty(new)
The guard condition changed from first_char == "[" to first_char == "[" || first_char == "{". Case 3 is now only reachable when jq is genuinely not installed, making its warning message accurate.
Error handling: All jq calls use 2>/dev/null and || true to suppress parse errors on malformed JSON. If jq parses successfully but finds no .result or .content, a descriptive placeholder is written with a warn() directing the user to inspect the raw JSON file.
-
Normalize all output to arrays before parsing — Pros: Single jq expression for both shapes, simpler control flow / Cons: Requires an extra jq invocation or
sedtransformation on every iteration; changes the file on disk which could affect downstream consumers; more complex error handling if the normalization itself fails on malformed input. -
Use a single jq expression with
if type == "array"conditional — Pros: One jq call instead of branching in bash / Cons: The jq expression becomes harder to read and debug (if type == "array" then .[-1].result else .result end // empty); error messages lose specificity about which shape was encountered; conflates two distinct extraction semantics into one expression. This approach was also observed in failure patterns — complex jq expressions on varied input types caused parse errors in production. -
Always dump raw JSON, skip extraction entirely — Pros: Zero parsing risk / Cons: Defeats the purpose of the function; iteration logs would contain raw JSON blobs instead of human-readable text summaries; downstream consumers (progress.md, dashboard) expect plain text.
- Files to create: none
- Files to modify:
-
scripts/sw-loop.sh—_extract_text_from_jsonfunction (lines 575-617): extend Case 2 guard, add object-specific jq expressions -
scripts/sw-loop-test.sh— Test 23: add 4 assertions for JSON object extraction and absence of misleading warning
-
- Dependencies: none (jq is an existing optional dependency)
- Risk areas:
- Regression in array handling: The array path is unchanged code, but sharing the Case 2 entry point means a typo in the guard condition could break it. Mitigated by existing Test 21 covering array extraction.
-
Malformed JSON objects: A file starting with
{but containing invalid JSON (e.g., truncated output) will fail jq parsing silently and fall through to the placeholder. This is correct behavior — same as existing array handling. -
Performance: No impact —
head -c1and thefirst_charbranch add negligible overhead.
- JSON object with
.resultfield: extracts result text, no warnings - JSON object with
.contentfield (no.result): extracts content as fallback - JSON object with neither field: writes descriptive placeholder, warns about missing
.result - JSON array input: behaves identically to before the change (regression check)
- "jq not available" warning does NOT appear when jq IS installed and input is
{...} - "jq not available" warning DOES appear when jq is genuinely absent
- All 69 existing loop tests continue to pass
-
Most likely: The
first_char == "["guard was written when Claude CLI only emitted JSON arrays. When the CLI started emitting single objects, the guard became an incomplete check, causing valid JSON to bypass jq. Evidence: The diff shows only the guard condition and branching within Case 2 were changed — the function structure was otherwise correct. -
Less likely: Claude CLI behavior changed in a version update, producing objects where arrays were expected. Evidence: Historical failure patterns in
failures.jsondocument both array and object outputs, suggesting both shapes have been possible for some time. - Unlikely: jq itself fails on object input. Evidence: jq handles objects natively; the issue was never reaching the jq call.
-
scripts/sw-loop.sh:579— original guard:first_char == "["(only arrays) -
scripts/sw-loop.sh:609— Case 3 warning triggered for objects:warn "JSON output but jq not available" - Memory
failures.json— 3 entries documenting jq parse errors on object-shaped output, confirming objects reach production - The fix: 18 lines changed in
sw-loop.sh, 45 lines added insw-loop-test.sh
The fix addresses the root cause (incomplete guard condition) rather than the symptom (misleading warning text). Changing only the warning message would hide the real issue — valid JSON objects not being parsed, causing raw JSON in iteration logs.
- Run
./scripts/sw-loop-test.sh— all 69 tests pass including 4 new assertions - Test 23 explicitly asserts the misleading warning is absent when jq is present
- Existing Test 21 covers array extraction as a regression gate