Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Pipeline Design 370

ezigus edited this page Apr 21, 2026 · 1 revision

ADR-012: Memory Recall and Store Integration in stage_intake

Context

Problem Statement: The Shipwright pipeline classifies GitHub issues in stage_intake() but currently does not leverage historical context about similar issues. This causes downstream stages (stage_plan(), stage_design()) to re-query the same classification data, creating redundant memory operations and missing opportunities for context enrichment at the earliest injection point.

Constraint:

  • The pipeline already uses ruflo memory in stage_plan() and stage_design() (lines 374–386, 910–922)
  • Issue #370 requires adding memory recall before those stages
  • Must maintain graceful degradation when ruflo is unavailable
  • Cannot introduce new external dependencies
  • Classification happens at line ~100 in stage_intake() via skill analysis

Opportunity: Intake stage is the first place where issue metadata (type, severity, labels) becomes available. This is the ideal injection point for historical context lookup, feeding all downstream stages.


Decision

Chosen Approach: Add sequential memory recall and store operations to stage_intake() following the exact guard pattern and call structure already established in stage_plan() and stage_design().

Component Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│ INTAKE STAGE (scripts/lib/pipeline-stages-intake.sh) │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ 1. Issue Classification (skill_analyze_issue) │
│ └─ Outputs: INTELLIGENCE_ISSUE_TYPE, INTELLIGENCE_SEVERITY │
│ ISSUE_LABELS │
│ │
│ 2. ┌─ Memory Recall Component ───────────────────────────────────────┐ │
│ │ ruflo_recall_similar_outcomes(type, labels) │ │
│ │ ├─ Guard: declare -f && ruflo_available │ │
│ │ └─ Output: _ruflo_intake_ctx (plain text, ~500-2000 bytes) │ │
│ │ └─ Export: INTELLIGENCE_INTAKE_CTX │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ 3. ┌─ Memory Store Component ─────────────────────────────────────┐ │
│ │ ruflo_store(key, value, namespace) │ │
│ │ ├─ Guard: declare -f │ │
│ │ ├─ Key: "stage-intake-result" │ │
│ │ ├─ Value: Classification summary │ │
│ │ └─ Namespace: "pipeline-${SHIPWRIGHT_PIPELINE_ID}" │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ 4. Save Artifact (intake.json) │
│ │
└─────────────────────────────────────────────────────────────────────────┘
 │ │
 └─────────────────────────┬──────────────────────────┐
 ↓ ↓
 stage_plan() stage_design()
 (consume (consume
 INTELLIGENCE_ INTELLIGENCE_
 INTAKE_CTX) INTAKE_CTX)

Component Definitions

Component Responsibility Input Output Failure Mode
Intake Classification Extract and classify issue metadata (type, severity, labels) Issue GH API response INTELLIGENCE_ISSUE_TYPE, INTELLIGENCE_SEVERITY, ISSUE_LABELS Returns defaults ("general", "unknown") — never fails
Memory Recall Query ruflo for similar historical outcomes Type + labels string INTELLIGENCE_INTAKE_CTX (plain text context) Returns empty string if unavailable/timeout — graceful
Memory Store Persist classification for downstream/historical queries Classification summary Success/failure (ignored) Logged but non-blocking (|| true)
Context Export Make intake context available to child processes INTELLIGENCE_INTAKE_CTX env var Exported to parent/child shell Safe default (empty string if unset)

Interface Contracts

ruflo_recall_similar_outcomes()

# Function signature
ruflo_recall_similar_outcomes(issue_type, labels) -> string
# Preconditions:
# - issue_type: non-empty string (e.g., "backend", "feature", "bug")
# - labels: comma-separated string, may be empty (e.g., "security,urgent" or "")
# - ruflo_available() must return 0 (caller responsibility to check)
# Postconditions:
# - stdout: Plain text context summary OR empty string
# - exit code: Always 0 (fail-open)
# - No side effects on stderr (errors suppressed with 2>/dev/null)
# Error Contract:
# - Timeout (30s max): Returns empty string, logged to debug
# - Network failure: Returns empty string, logged to debug
# - No matches found: Returns empty string (valid outcome)
# - NEVER propagates errors to caller
# Invocation Pattern (Intake):
local ctx; ctx=$(ruflo_recall_similar_outcomes "${INTELLIGENCE_ISSUE_TYPE:-general}" "${ISSUE_LABELS:-}" 2>/dev/null || true)
if [[ -n "$ctx" ]]; then
 INTELLIGENCE_INTAKE_CTX="$ctx"
 export INTELLIGENCE_INTAKE_CTX
fi

ruflo_store()

# Function signature
ruflo_store(key, value, namespace) -> exit_code
# Preconditions:
# - key: non-empty string (e.g., "stage-intake-result")
# - value: plain text <500 chars, e.g., "Issue type: backend. Severity: critical."
# - namespace: non-empty string (e.g., "pipeline-abc123")
# - No size limits enforced by this component (caller validates)
# Postconditions:
# - Data stored in ruflo memory under namespace
# - exit code: 0 on success, non-zero on failure
# - Side effects: Logged to pipeline artifacts
# Error Contract:
# - Storage failure: Returns non-zero, error logged (non-blocking)
# - Invalid parameters: Caller responsibility to validate (pre-check)
# - NEVER blocks pipeline execution (all calls use || true)
# Invocation Pattern (Intake):
local summary="Issue type: ${INTELLIGENCE_ISSUE_TYPE:-unknown}. Severity: ${INTELLIGENCE_SEVERITY:-unknown}. Labels: ${ISSUE_LABELS:-none}."
ruflo_store "stage-intake-result" "$summary" "pipeline-${SHIPWRIGHT_PIPELINE_ID:-unknown}" || true

INTELLIGENCE_INTAKE_CTX (Exported Environment Variable)

# Variable: INTELLIGENCE_INTAKE_CTX
# Type: String (plain text, unstructured, <2000 bytes)
# Scope: Exported from stage_intake, consumed by stage_plan/stage_design
# Lifetime: Duration of pipeline execution
# Default: Empty string (unset)
# Format: Plain text context from ruflo_recall_similar_outcomes
# Example value:
# "Similar issue: Auth module rewrite (3 days). Affected users: ~1000.
# Historical severity: critical. Recommended: spike + design review."
# Consumption Pattern (plan/design):
if [[ -n "${INTELLIGENCE_INTAKE_CTX:-}" ]]; then
 prompt="${prompt}\n## Intake Context\n${INTELLIGENCE_INTAKE_CTX}"
fi
# Guarantees:
# - Always defined (unset = ""), never cause parameter expansion errors
# - Read-only within downstream stages (no modifications)
# - Available to all child processes of pipeline

Data Flow

GitHub Issue (API response)
 │
 ├─ title, labels, body, assignee, milestone
 │
 ▼
┌─────────────────────────┐
│ skill_analyze_issue() │
└─────────────────────────┘
 │
 ├─ INTELLIGENCE_ISSUE_TYPE (e.g., "backend", "ui-test")
 ├─ INTELLIGENCE_SEVERITY (e.g., "critical", "low")
 └─ ISSUE_LABELS (e.g., "feature,urgent")
 │
 ▼
┌──────────────────────────────────────────────┐
│ Memory Recall │
│ Input: type + labels │
└──────────────────────────────────────────────┘
 │
 ▼ (vector similarity search in ruflo)
 │
 ├─ Similar past outcomes from same issue type
 ├─ Historical patterns (time estimates, risks)
 └─ Domain context from previous issues
 │
 ▼
INTELLIGENCE_INTAKE_CTX (plain text summary)
 │
 ├─ Exported to parent/child shell
 │
 ├──────────────────────────────────────────────────┐
 │ │
 ▼ ▼
stage_plan() prompt injection stage_design() prompt injection
 │ │
 └──────────────────────────────────────────────────┘
 │
 ▼
 Enriched planning/design
 with intake context

Error Boundaries

Layer Component Error Type Handling Propagation Recovery
Intake Classify Issue API fails Try catch, defaults Pipeline halts User retries issue
Intake Recall Function undefined declare -f check Silently skip Proceed without context
Intake Recall ruflo unavailable ruflo_available() check Silently skip Proceed without context
Intake Recall Timeout/network 2>/dev/null || true Returns empty string Use empty context (safe)
Intake Store Function undefined declare -f check Silently skip No storage (acceptable)
Intake Store Storage failure || true Logged, continues Downstream ignores (safe)
Plan Consume INTELLIGENCE_INTAKE_CTX unset [[ -n "$var" ]] guard Uses default Proceeds with empty context
Design Consume INTELLIGENCE_INTAKE_CTX unset [[ -n "$var" ]] guard Uses default Proceeds with empty context

Key Principle: All errors are non-blocking. The pipeline never halts due to memory operations.


Alternatives Considered

1. Deferred Recall in Downstream Stages Only (Rejected)

Approach: Skip intake recall; let stage_plan() and stage_design() continue calling ruflo_recall_similar_outcomes() independently.

Pros:

  • No changes to intake stage (minimal)
  • Each stage queries fresh context

Cons:

  • ✗ Violates requirement: "earliest injection point"
  • ✗ Redundant queries (plan AND design each call recall) — wasteful
  • ✗ Missing opportunity to enrich intake artifact with context
  • ✗ Defeats purpose: context enrichment at source

Why Rejected: Directly contradicts issue goal of adding memory to intake. Also wasteful (duplicate queries).


2. Separate Memory Initialization Pipeline Stage (Rejected)

Approach: Create a new stage stage_memory_intake between intake and plan that handles all memory operations.

Pros:

  • Clear separation of concerns (classification vs. memory)
  • Explicit stage for memory initialization

Cons:

  • ✗ Increases pipeline complexity (+1 stage, +1 gate, +1 state transition)
  • ✗ Breaks established pattern (recall/store happen inside stages, not between them)
  • ✗ Requires new stage testing, new hooks, new artifact handling
  • ✗ Overkill for 2 function calls (violates principle: don't add abstraction for one-off uses)
  • ✗ Harder to debug (memory state isolated from classification context)

Why Rejected: Unnecessary abstraction. Established pattern is to recall/store within stages, not create wrapper stages.


3. Batch Recall at Pipeline Start (Rejected)

Approach: Pre-fetch all memory context for all potential issue types in pipeline_start() before any stage runs.

Pros:

  • Centralized memory operations
  • Could cache results

Cons:

  • ✗ Requires knowing all issue types ahead of time (we don't)
  • ✗ Fetches context that may not be used
  • ✗ Cannot batch-load without issue classification first (circular)
  • ✗ Increases startup latency

Why Rejected: Requires classification before recall, but classification happens inside intake. Creates circular dependency.


4. Inline Memory Operations Without Guards (Rejected)

Approach: Simplify by removing declare -f and ruflo_available() checks, assume ruflo always present.

Pros:

  • Simpler code (fewer lines)

Cons:

  • ✗ Pipeline fails if ruflo unavailable (not backwards compatible)
  • ✗ Violates defensive programming principle
  • ✗ Cannot run pipeline in environments without ruflo (CI, local dev)
  • ✗ Breaks existing behavior of other stages that already guard

Why Rejected: Breaks existing pattern. Established code (stage_plan, stage_design) already uses guards—we must match.


Implementation Plan

Files to Modify

  1. scripts/lib/pipeline-stages-intake.sh (4 insertions, 0 deletions)

    • Add ruflo_recall_similar_outcomes() call after line 108 (after skill analysis)
    • Add ruflo_store() call after line 95 (after artifact save)
    • Update stage_plan() to consume INTELLIGENCE_INTAKE_CTX (after line 407)
    • Update stage_design() to consume INTELLIGENCE_INTAKE_CTX (after line 922)
  2. scripts/sw-lib-pipeline-stages-test.sh (5 new test functions)

    • Test stage_intake with ruflo available → INTELLIGENCE_INTAKE_CTX exported
    • Test stage_intake with ruflo unavailable → graceful no-op
    • Test ruflo_store() called with correct namespace
    • Test stage_plan() consumes context in prompt
    • Test stage_design() consumes context in prompt

Files to Create

None — all integration uses existing test file.

New Dependencies

None — uses existing ruflo interface (already imported in pipeline-stages-intake.sh).

Risk Areas

Risk Severity Mitigation
declare -f guard missing or typo Medium Review all 4 declare -f statements; ensure exact pattern matches stage_plan/design
INTELLIGENCE_INTAKE_CTX not exported Medium Test case verifies export; run env | grep INTELLIGENCE in test
Downstream stages don't consume context Medium Update both stage_plan and stage_design in same commit; test both
ruflo_store() fails silently Low Non-blocking by design; log messages show attempt occurred
Variable name collision Low Search codebase for INTELLIGENCE_INTAKE_CTX; confirm not used elsewhere
Merge conflicts in stage_plan/design Low Make changes in clearly separate sections; use multi-line comments to anchor changes

Validation Criteria

Acceptance Criteria (from issue #370)

  • ruflo_recall_similar_outcomes() is called inside stage_intake() after issue classification (line ~108)
  • Result is exported as INTELLIGENCE_INTAKE_CTX for downstream consumption
  • ruflo_store("stage-intake-result", ...) is called with classification summary
  • Both calls are guarded with declare -f checks
  • Both calls use || true — never block pipeline
  • Passes ./scripts/sw-pipeline-test.sh (full pipeline execution)
  • Passes npm test (no regressions)
  • Graceful no-op when ruflo_available() returns false

Design Quality Criteria

  • Code follows exact pattern from stage_plan() lines 374–386 (guard structure, variable names, error handling)
  • Code follows exact pattern from stage_design() lines 910–922 (same)
  • Variable naming matches convention: _ruflo_intake_ctx (local), INTELLIGENCE_INTAKE_CTX (exported)
  • Guard structure matches: if declare -f ... && ... && ruflo_available; then
  • Error messages are consistent with pipeline logging (info "...", warn "...")
  • No new external dependencies introduced
  • File organization preserved — all changes in scripts/lib/pipeline-stages-intake.sh only

Test Coverage Criteria

  • Unit test: stage_intake with ruflo_available=true → exports INTELLIGENCE_INTAKE_CTX
  • Unit test: stage_intake with ruflo_available=false → no-op, no export ✓
  • Unit test: stage_intake calls ruflo_store() with correct namespace pipeline-{ID}
  • Unit test: stage_plan() appends INTELLIGENCE_INTAKE_CTX to prompt when set ✓
  • Unit test: stage_design() appends INTELLIGENCE_INTAKE_CTX to prompt when set ✓
  • Integration test: Full pipeline intake → plan → design with context propagation ✓
  • Regression test: All existing tests in sw-lib-pipeline-stages-test.sh pass ✓

Runtime Verification

# After implementation, verify:
# 1. intake stage exports the variable
SHIPWRIGHT_PIPELINE_ID=test-123 stage_intake
[[ -n "$INTELLIGENCE_INTAKE_CTX" ]] && echo "✓ Context exported" || echo "✗ FAIL"
# 2. Graceful handling when ruflo absent
unset -f ruflo_recall_similar_outcomes
SHIPWRIGHT_PIPELINE_ID=test-123 stage_intake
echo "✓ No error despite missing function" || echo "✗ FAIL"
# 3. stage_plan consumes context
grep -q "Intake Context" "$ARTIFACTS_DIR/plan.md" && echo "✓ Injected into plan" || echo "✗ FAIL"
# 4. Full pipeline test
./scripts/sw-pipeline-test.sh

Rationale

Why intake is the right place:

  • Issue classification produces INTELLIGENCE_ISSUE_TYPE, INTELLIGENCE_SEVERITY, ISSUE_LABELS at line ~100
  • This is the earliest available context for vector similarity search
  • All downstream stages (plan, design, build, test) benefit immediately without redundant queries

Why follow existing patterns:

  • stage_plan() and stage_design() already use ruflo_recall_similar_outcomes() (lines 374–386, 910–922)
  • Our changes must match this pattern exactly (guardrails, error handling, variable names)
  • Consistency reduces cognitive load and review friction

Why non-blocking error handling:

  • Pipeline must never halt due to memory operations (memory is optimistic enhancement, not critical path)
  • Use declare -f guards to check function availability before calling
  • Use || true to prevent exit code propagation
  • Use 2>/dev/null to suppress error output

Why export INTELLIGENCE_INTAKE_CTX:

  • Must flow to downstream stages (plan, design, build) for context use
  • Environment variable is the natural IPC mechanism in shell pipelines
  • Guard downstream consumption with [[ -n "$var" ]] to handle unset case

Summary

This ADR documents a minimal, pattern-preserving change to inject historical memory context at the earliest point in the pipeline where issue classification occurs. The design:

  1. ✓ Adds 2 ruflo operations (recall + store) in proper sequence
  2. ✓ Follows established patterns from stage_plan/stage_design
  3. ✓ Implements graceful degradation (works without ruflo)
  4. ✓ Exports context for downstream consumption
  5. ✓ Maintains zero new dependencies
  6. ✓ Includes comprehensive test coverage

The implementation is low-risk, high-value: minimal code surface, proven patterns, clear error boundaries, and immediate benefit to all downstream stages.

Clone this wiki locally

AltStyle によって変換されたページ (->オリジナル) /