Shipwright has 22 unit/component test suites that mock external dependencies (Claude API, GitHub API, tmux) but zero integration tests that exercise the real pipeline orchestration end-to-end. This means regressions in stage sequencing, state file management, and cross-script coordination go undetected until manual testing.

Constraints from the codebase:

• All scripts are Bash 3.2 compatible (set -euo pipefail, no associative arrays, no readarray)
• Test harness convention: PASS/FAIL counters, ERR trap, colored output via info()/success()/error() helpers
• Pipeline state lives in .claude/pipeline-state.md and .claude/pipeline-artifacts/
• Existing CI (test.yml) runs npm test which executes all 22 test suites via package.json scripts
• Pipeline templates are JSON files in templates/pipelines/ — the pipeline reads them to determine which stages to run, iteration counts, and gating
• Budget enforcement exists in sw-cost.sh — the pipeline checks remaining budget before each stage
• --dry-run flag is already supported by sw-pipeline.sh and skips actual Claude invocations
• $NO_GITHUB env var disables all GitHub API calls throughout the codebase

Two-tier integration test architecture with clear isolation between deterministic smoke tests (Tier 1) and API-dependent live tests (Tier 2).

A new script scripts/sw-integration-test.sh following the existing test harness pattern. It exercises real pipeline orchestration with mock Claude/GitHub binaries — the same mocking approach used in sw-pipeline-test.sh but focused on end-to-end flow rather than individual function behavior.

Four smoke test cases:

1. Dry-run smoke — sw pipeline start --goal "test" --dry-run exits 0, emits "Dry run" on stdout, creates .claude/ directory structure
2. Stage ordering — A mocked pipeline runs the integration template (intake → build → test), verifying each stage executes in sequence by checking ordered timestamps in the state file
3. State file integrity — After a pipeline run, validates .claude/pipeline-state.md contains required fields (stage names, timestamps, status per stage, goal)
4. Budget enforcement — Sets budget to 0ドル.00, runs pipeline, verifies it exits cleanly (exit 0 or well-defined exit code) with a budget-exceeded message rather than crashing

Mock strategy: Each test case creates a temp directory, populates it with mock binaries for claude and gh (echoing expected output), sets PATH to prefer mocks, sets NO_GITHUB=1, and runs the pipeline. This is identical to the pattern in sw-pipeline-test.sh:45-80 where mock binaries are set up.

Three live test cases that call the real Claude API with strict budget controls:

5. README modification — Creates a temp git repo, runs sw pipeline start --goal "Add a one-line description to README.md" --template integration, verifies git diff shows README changes
6. PR creation — Runs a full fast template pipeline against the temp repo, verifies a PR branch exists and the working tree is clean
7. Budget cap verification — After the live run, reads ~/.shipwright/costs.json and asserts total spend is under 1ドル.00

Safety mechanisms:

• SHIPWRIGHT_BUDGET_LIMIT=1.00 environment variable hard-caps spending
• 15-minute job timeout in CI prevents runaway API calls
• Tests run against a throwaway temp repo (not the real Shipwright repo)
• INTEGRATION_LIVE must be explicitly set — accidental runs impossible

A new templates/pipelines/integration.json with minimal stages:

• Stages: intake → build → test (3 stages only)
• Model: sonnet (cheapest capable model)
• Max iterations: 3 (enough to verify the loop, cheap enough to cap costs)
• All gates: auto (no human approval needed)
• No PR/deploy/monitor stages — keeps costs and complexity minimal

A new .github/workflows/integration-test.yml with three jobs:

Each job writes per-test-case results to $GITHUB_STEP_SUMMARY as a markdown table (test name, status, duration).

The existing test.yml gets a new parallel job integration-smoke that runs npm run test:integration alongside the existing unit test job — ensuring smoke tests block PRs just like unit tests do.

PR opened
 ├── test.yml → unit tests (existing 22 suites)
 │ → integration-smoke (NEW: Tier 1)
 └── integration-test.yml
 ├── smoke job (Tier 1 — always)
 ├── live job (Tier 2 — when API key available)
 └── regression job (Tier 2 — main branch only)

• Smoke tests: ERR trap captures failures, logs the failing test case, increments FAIL counter, continues to next test. Final exit code = 1 if any FAIL > 0.
• Live tests: Same ERR trap pattern. Additionally, if the budget check fails mid-run, the test captures the exit status and verifies it's the expected budget-exceeded code (not a crash).
• CI: Job-level timeout-minutes prevents infinite hangs. continue-on-error: false on smoke jobs means they block merge. Live jobs use continue-on-error: true initially (since API key may not be configured in all forks).

1. Extend sw-pipeline-test.sh with integration cases — Pros: No new file, reuses existing mock setup. / Cons: Mixes unit-level function tests with end-to-end flow tests, making failures harder to diagnose. The existing file is already 1757 lines. Integration tests have fundamentally different setup (full temp repo vs. function-level mocking) and different CI characteristics (Tier 2 needs secrets, longer timeouts).

2. Use a testing framework (bats-core, shunit2) — Pros: Structured test discovery, TAP output, better assertion primitives. / Cons: Introduces a new dependency not used anywhere else in the project. All 22 existing test suites use the custom PASS/FAIL harness pattern. Adopting a framework for one suite creates inconsistency and requires all contributors to learn a new tool. The custom harness is simple and well-understood.

3. Docker-based integration tests — Pros: Perfect isolation, reproducible environment, no host contamination. / Cons: Adds Docker as a CI dependency, increases build time significantly, complicates debugging. The temp-directory + mock-binary approach already provides sufficient isolation without the overhead. Shipwright targets macOS developers — Docker adds friction.

4. Single-tier approach (smoke only, no live tests) — Pros: Simpler, no API costs, no secrets management. / Cons: Misses the highest-value validation — that the pipeline actually produces correct output when talking to a real LLM. The tiered approach gives us both: fast deterministic feedback on every PR + real validation when API access is available.

• Files to create:

  • scripts/sw-integration-test.sh — Main integration test script (~300-400 lines)
  • templates/pipelines/integration.json — Minimal pipeline template (~30 lines)
  • .github/workflows/integration-test.yml — CI workflow (~80 lines)

• Files to modify:

  • package.json — Add test:integration and test:integration:live scripts
  • .github/workflows/test.yml — Add integration-smoke parallel job
  • .claude/CLAUDE.md — Add test suite 23 to the test suites table and update count

• Dependencies: None. Uses only existing tools (bash, jq, git, gh).

• Risk areas:

  • Live test flakiness: Claude API responses are non-deterministic. Tier 2 tests should assert structural properties (file changed, PR exists, cost under cap) not exact content. Retry logic with --max-retries 1 for transient API failures.
  • Cost creep: If the integration template or iteration count is accidentally increased, live test costs could spike. The 1ドル.00 budget hard-cap in CI environment variable is the safety net, but the template itself should also specify max_cost: 1.0.
  • State file format changes: If sw-pipeline.sh changes the state file format, Tier 1 test 3 (state file integrity) will break. Mitigate by testing for structural properties (has timestamps, has stage names) rather than exact field positions.
  • Mock binary drift: If sw-pipeline.sh starts calling new external tools that aren't mocked, smoke tests will fail with "command not found." This is actually desirable — it surfaces new dependencies early.
  • CI secret availability: Forks won't have ANTHROPIC_API_KEY. The live job must gracefully skip (not fail) when the secret is absent. Use if: secrets.ANTHROPIC_API_KEY != '' in the workflow.

• ./scripts/sw-integration-test.sh exits 0 with all PASS, no FAIL — no API key needed
• INTEGRATION_LIVE=true ./scripts/sw-integration-test.sh runs both tiers when ANTHROPIC_API_KEY is set
• npm run test:integration executes smoke tests and exits 0
• npm run test:integration:live executes both tiers and exits 0 (when API key present)
• Existing npm test still passes — no regression in the 22 existing suites
• CI smoke job runs on PRs without secrets and blocks merge on failure
• CI live job skips gracefully when ANTHROPIC_API_KEY is not configured
• Live tests complete within 15 minutes and under 1ドル.00 total spend
• State file validation catches missing fields (test with intentionally malformed state)
• Budget enforcement test confirms clean exit (not crash/unhandled error) at 0ドル budget
• $GITHUB_STEP_SUMMARY shows per-test markdown table in CI
• templates/pipelines/integration.json is valid JSON and loadable by sw-pipeline.sh