-
Notifications
You must be signed in to change notification settings - Fork 1
Pipeline Design 232
Now I have enough context. Let me produce the ADR.
Failed pipelines waste ~5ドル/run and ~4133s build time (per recent metrics) because broken test harnesses are only discovered late in the build-test loop. Root causes from recent failures (failures.json, 2026年03月09日) include missing artifact writes and jq parse errors — exactly the kind of issues a pre-flight check would catch early.
The Shipwright bash test harness has two valid patterns:
-
Inline:
set -euo pipefail+PASS=0/FAIL=0+trap ... ERRdirectly in the test file -
Sourced:
source "$SCRIPT_DIR/lib/test-helpers.sh"which provides counters, traps, and assertions automatically
Both patterns must be recognized as valid. Non-bash projects (Node/pytest/Cargo) need only lightweight validation (test command resolves, test directory exists).
Constraints:
- Bash 3.2 compatible (no
declare -A, noreadarray) - Must not break existing working pipelines
- Must handle new projects with no test files yet
- shellcheck is optional (not all environments have it)
Create scripts/lib/pipeline-test-validation.sh as a guard-loaded library module following the exact pattern of pipeline-quality-checks.sh and pipeline-detection.sh. The module provides validate_test_infrastructure() called from stage_intake() after step 3 (test command auto-detection, line 65) and before step 4 (branch creation, line 68).
stage_intake()
→ detect_test_cmd() # step 3 — sets TEST_CMD
→ validate_test_infrastructure() # NEW — reads TEST_CMD, PROJECT_ROOT
→ find_test_files() # discovers *-test.sh, *.test.ts, etc.
→ validate_bash_test_harness() # per-file .sh validation
→ optional shellcheck pass
→ _write_validation_report() # JSON to pipeline-artifacts/
→ branch creation # step 4 — unchanged
Severity model: ERROR blocks intake (return 1). WARNING is logged but non-blocking. INFO is informational.
| Check | Scope | Severity | Detail |
|---|---|---|---|
| TEST_CMD binary resolves | All projects | ERROR | command -v $(echo "$TEST_CMD" | awk '{print 1ドル}') |
| Test files discovered | All projects | INFO | Missing = warning for new projects, not an error |
| File is executable | .sh test files | ERROR | -x "$file" |
set -euo pipefail or set -e
|
.sh test files | ERROR | grep -qE 'set -e(uo pipefail)?' |
| PASS/FAIL counters present | .sh test files | ERROR | Either inline PASS=0 or source.*test-helpers
|
| ERR trap present | .sh test files | WARNING |
grep -qE 'trap.*ERR' (test-helpers.sh provides this) |
bash -n syntax check |
.sh test files | ERROR | Catches parse errors before runtime |
| shellcheck | .sh test files | WARNING | Only when command -v shellcheck succeeds |
Key design decision: When a .sh test file sources test-helpers.sh, the PASS/FAIL counter and ERR trap checks are automatically satisfied — the validator recognizes this pattern and skips those individual checks.
-
validate_test_infrastructure()returns 0 (pass) or 1 (errors found) - On return 1,
stage_intake()emitsintake.test_validation_failedevent and returns 1 -
SKIP_TEST_VALIDATION=truebypasses entirely with an info message - No test files found = INFO-level message, return 0 (new project safe)
- shellcheck missing = INFO-level skip, not a failure
- JSON report always written (even on skip —
"skipped": true)
Written to $ARTIFACTS_DIR/test-validation.json via _write_validation_report() using jq -n --arg for safe JSON construction (no string interpolation):
{
"timestamp": "2026年03月09日T12:00:00Z",
"valid": true,
"test_cmd": "npm test",
"test_files_found": 5,
"checks": [
{"file": "scripts/sw-hello-test.sh", "check": "executable", "severity": "error", "passed": true, "message": "File is executable"}
],
"errors": 0,
"warnings": 1,
"skipped": false,
"shellcheck_available": true
}-
Inline in stage_intake() — Pros: no new files, zero module overhead. Cons:
stage_intake()already 119 lines, mixes concerns, harder to unit test independently. Would violate the separation pattern established bypipeline-detection.sh/pipeline-quality-checks.sh. -
Extend pipeline-detection.sh — Pros: colocates with
detect_test_cmd(). Cons: detection (what) vs validation (is it correct) are different concerns.pipeline-detection.shis sourced by many modules; growing it increases blast radius. Adding ~150 lines of validation logic would make it the largest lib module. -
Standalone script (sw-test-validation.sh) — Pros: runnable independently. Cons: can't share globals (
TEST_CMD,ARTIFACTS_DIR), would need parameter passing, breaks the library module pattern used by all other pipeline components.
-
scripts/lib/pipeline-test-validation.sh(~150 lines) — module guard,find_test_files(),validate_bash_test_harness(),validate_test_infrastructure(),_write_validation_report() -
scripts/sw-test-validation-test.sh(~300 lines) — full test suite sourcingtest-helpers.sh
-
scripts/lib/pipeline-cli.sh(line ~141) — add--skip-test-validation) SKIP_TEST_VALIDATION=true; shift ;;toparse_args() -
scripts/lib/pipeline-stages-intake.sh(between lines 65-67) — callvalidate_test_infrastructure()after test command detection, before branch creation -
scripts/lib/pipeline-stages.sh(after line 190) — sourcepipeline-test-validation.sh -
package.json— add"test:test-validation": "bash scripts/sw-test-validation-test.sh"to scripts
- None new. Uses existing
jq,bash -n, optionalshellcheck.
-
False positives on non-Shipwright projects: Mitigated by only enforcing bash harness checks on
.shfiles. Node/Python/Rust/Go projects get lightweight validation only (test command resolves, test directory exists). - shellcheck variability: Different shellcheck versions produce different warnings. Using WARNING severity means this never blocks.
-
bash -nfalse negatives:bash -ncatches syntax errors but not runtime errors. This is intentional — it's a fast pre-flight check, not a full test runner. - Insertion point in stage_intake(): Between step 3 and step 4 is safe because test command detection (step 3) has no side effects, and branch creation (step 4) is the first side effect. Failing before branch creation means no cleanup needed.
- Pipeline with valid Shipwright test files (both inline and test-helpers.sh patterns) passes intake validation
- Pipeline with a non-executable .sh test file fails at intake with "not executable" error message
- Pipeline with a .sh test file missing
set -euo pipefailfails at intake - Pipeline with a .sh test file that sources
test-helpers.shpasses (PASS/FAIL provided externally) - Pipeline with
--skip-test-validationbypasses all checks and logs info message -
test-validation.jsonwritten to$ARTIFACTS_DIRwith correct schema - shellcheck runs when installed, skips with INFO when not installed
- No test files found (new project) produces INFO warning, returns 0
-
bash -ncatches syntax errors in test files - Non-bash projects (Node, Python, Go) get lightweight validation (TEST_CMD resolves) without bash harness checks
- No regressions in
sw-pipeline-test.sh(existing 58 tests pass) - Test suite
sw-test-validation-test.shregistered inpackage.jsonand passes with 0 failures
Unit tests (15 tests, ~70%): Each validation function tested in isolation:
-
find_test_files(): discovers .sh tests, discovers .test.ts files, handles empty project, filters non-test files -
validate_bash_test_harness(): passes valid inline harness, passes test-helpers sourced harness, fails non-executable, fails missing set -e, fails missing PASS/FAIL, fails bash -n syntax error, handles shellcheck available/unavailable -
_write_validation_report(): correct JSON schema, handles zero checks, handles errors vs warnings
Integration tests (4 tests, ~25%): Full validate_test_infrastructure() flow:
- Valid Shipwright project with mixed test patterns → passes
- Project with one broken test file → fails with error count
-
SKIP_TEST_VALIDATION=true→ bypasses, writes skipped report - Non-bash project (Node.js mock) → lightweight validation only
E2E test (1 test, ~5%):
- Broken test file in mock project →
stage_intake()returns 1 (requires mock ofghand git)
- 100% of the 8 check types (executable, set -e, PASS/FAIL, ERR trap, bash -n, shellcheck, test-helpers pattern, test command resolution)
- Both pass and fail paths for every ERROR-severity check
- Graceful degradation paths: shellcheck missing, no test files, new project
Happy path: Valid .sh test file with set -euo pipefail, PASS=0, FAIL=0, trap ... ERR → all checks pass, report shows "valid": true, "errors": 0
Error cases:
- Non-executable test file (chmod 644) → ERROR,
"valid": false, intake returns 1 - Test file with
#!/usr/bin/env bashbut missingset -euo pipefail→ ERROR on harness check - Test file with syntax error (unmatched quote) →
bash -ncatches it, ERROR severity
Edge cases:
- Test file sources
test-helpers.shbut has no inline PASS/FAIL → valid (sourced pattern recognized) -
TEST_CMD="npm test"butnpmnot on PATH → ERROR on command resolution (but this is unlikely in practice; may want WARNING instead) - Mixed project: some .sh tests valid, one invalid → reports per-file, overall
"valid": false -
SKIP_TEST_VALIDATION=truewith broken test files → bypassed, report has"skipped": true