-
Notifications
You must be signed in to change notification settings - Fork 1
Pipeline Plan 17
Seth Ford edited this page Feb 11, 2026
·
4 revisions
The plan has been fully developed. Here's the complete implementation plan:
Replace JSON file-based state management with SQLite for atomic concurrent writes, efficient aggregate queries, and schema migration support. Dual-write approach: events go to both JSONL and SQLite for backward compatibility. Falls back to JSON when sqlite3 is unavailable.
| File | Purpose |
|---|---|
scripts/lib/migrations/001_initial_schema.sql |
7 tables: events, pipeline_runs, sessions, developers, cost_entries, metrics, memory_entries + schema_version |
scripts/lib/migrations/002_indexes.sql |
Composite indexes on type+epoch, issue, stage, repo_hash |
scripts/lib/db.sh |
Bash SQLite helper library — sw_db_init(), sw_db_migrate(), sw_db_available(), insert/query wrappers |
dashboard/db.ts |
TypeScript module using bun:sqlite — typed query helpers replacing O(n) JS loops with indexed SQL |
scripts/sw-db.sh |
CLI subcommand: shipwright db migrate/status/export/import/query
|
scripts/sw-db-test.sh |
15+ test cases covering CRUD, migrations, concurrency, fallback |
| File | Change |
|---|---|
scripts/lib/helpers.sh |
Add dual-write db_emit_event() — writes both JSONL and SQLite |
scripts/sw-daemon.sh |
Track pipeline_runs in SQLite; SQL-based DORA metric queries |
scripts/sw-cost.sh |
Write cost_entries; SQL aggregation for summaries |
scripts/sw-connect.sh |
Upsert developers table from heartbeats |
scripts/sw-memory.sh |
Write memory_entries for patterns/failures |
dashboard/server.ts |
Use dashboard/db.ts for reads, JSON fallback |
scripts/sw |
Register db subcommand |
package.json |
Add sw-db-test.sh to test chain |
.claude/CLAUDE.md |
Document db subcommand and schema |
- Create migration SQL files (schema + indexes)
- Create
scripts/lib/db.sh— bash SQLite helpers - Create
dashboard/db.ts— TypeScript SQLite module - Modify
helpers.sh— dual-write event emission - Modify
sw-daemon.sh— pipeline_runs + SQL DORA queries - Modify
sw-cost.sh— SQLite cost recording - Modify
sw-connect.sh— SQLite developer registry - Modify
sw-memory.sh— SQLite memory entries - Modify
dashboard/server.ts— SQL reads with JSON fallback - Create
sw-db.sh— CLI subcommand - Register
dbinscripts/swrouter - Create
sw-db-test.sh— test suite - Update
package.json+ CLAUDE.md - Run full test suite, fix regressions
- Dual-write: Events written to both JSONL and SQLite — no migration cutover needed
- WAL mode: Concurrent dashboard reads while daemon writes
-
Zero dependencies:
bun:sqlitefor TypeScript,sqlite3CLI for bash (ships with macOS/Linux) -
JSON blob columns:
data TEXTon events/developers/memory for flexible fields alongside indexed columns -
Graceful degradation: Every SQLite read has a JSON fallback path —
sw_db_available()gates all SQL paths
The full detailed plan with SQL schemas, example queries, and test specifications is ready. The plan was written to .claude/pipeline-artifacts/plan.md.