Architecture

lacausecrypto edited this page Apr 6, 2026 · 3 revisions

Architecture

How OCC works under the hood.

System Overview

┌──────────────────┐ ┌────────────────────┐
│ Claude Code │──MCP──▶│ MCP Server │
│ Claude Desktop │ stdio │ (index.ts) │
└──────────────────┘ │ 28 tools exposed │
 └────────┬───────────┘
 │
┌──────────────────┐ ┌────────▼───────────┐ ┌──────────────────┐
│ curl / Browser │──HTTP──▶│ REST Server │◀──────│ React Dashboard │
│ Any client │ :4242 │ (rest.ts) │ :5173 │ (frontend-react)│
└──────────────────┘ │ 102 endpoints │ │ Canvas + Chat │
 │ SSE streaming │ └──────────────────┘
 └────────┬───────────┘
 │
 ┌───────────────────────────┼───────────────────────────┐
 ▼ ▼ ▼
 ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
 │ Loader │ │ Executor │ │ Scheduler │
 │ (loader.ts) │ │(executor.ts) │ │(scheduler.ts)│
 │ YAML parsing │ │ Chain runner │ │ Cron jobs │
 │ Zod validation│ │ Wave parallel │ │ Auto-execute │
 │ Dep. graph │ │ Checkpoints │ └──────────────┘
 └──────┬───────┘ └──────┬───────┘
 │ │
 ▼ ┌──────▼───────┐
 ┌──────────┐ │ Providers │
 │ chains/ │ │(providers.ts) │
 │ (YAML) │ │ Claude CLI │
 └──────────┘ │ OpenRouter │
 │ OpenAI │
 ┌──────────┐ │ Ollama │
 │ SQLite DB │◀──────────────│ HuggingFace │
 │ (storage) │ checkpoints │ Custom │
 │ (queue) │ executions └───────────────┘
 └──────────┘

Module Breakdown (18 modules)

Core

Module	Description
`rest.ts`	Express HTTP server — 102 endpoints, SSE streaming, CORS, security headers
`index.ts`	MCP server — 28 tools over stdio transport
`executor.ts`	Chain execution engine — wave parallelism, checkpoints, cancellation
`loader.ts`	YAML loading, Zod validation, dependency graph (topological sort)
`pipeline-loader.ts`	Pipeline YAML loading and validation
`pipeline-executor.ts`	Pipeline orchestration (chain-level dependency graph)

Infrastructure

Module	Description
`storage.ts`	SQLite persistence — executions, step checkpoints, WAL mode
`queue.ts`	SQLite job queue — priority scheduling, worker pool, retry
`scheduler.ts`	Cron-based chain scheduling with `node-cron`
`providers.ts`	Multi-LLM provider management — 6 provider types, AES-256-GCM key encryption
`claude-runner.ts`	Claude CLI subprocess management — spawn, stream, timeout, kill
`mcp-client.ts`	External MCP server client — connect, discover tools, call

Tools

Module	Description
`pretool-executor.ts`	29 pre-tool types execution (http_fetch, bash, vector_query, etc.)
`pretool-extras.ts`	Advanced pre-tools: state management, vector DB, graph DB, semantic cache
`linter.ts`	Chain linter — 20+ lint rules, dry-run execution planner
`style-extractor.ts`	CSS/color extraction from URLs (Playwright headless for CSS-in-JS)
`blob.ts`	BLOB canvas — organic workflow graph with autonomous agent loop
`logger.ts`	Structured logging with level filtering

loader.ts — Chain Loading & Validation

YAML parsing — js-yaml converts YAML to JavaScript objects
Zod validation — Every field validated against strict schema
Dependency graph — Topological sort (Kahn's algorithm) produces execution waves
Path resolution — Uses fileURLToPath for cross-platform compatibility (handles spaces in paths)

Chain YAML → js-yaml → Zod safeParse → buildDependencyGraph → waves[]

Waves are groups of steps with no dependencies between them:

Wave 1: [step_a, step_b, step_c] ← parallel
Wave 2: [step_d] ← depends on a,b
Wave 3: [step_e, step_f] ← depends on d

executor.ts — Chain Execution Engine

The core execution loop:

for each wave in dependency_graph:
 await Promise.all(
 wave.map(step => executeStep(step))
 )

Step execution flow:

1. Check condition → skip if false
2. Check cache → return cached if valid
3. Run pre-tools (29 types) → inject variables
4. Resolve {variables} in prompt
5. Route to LLM provider (Claude CLI / OpenRouter / OpenAI / Ollama / HuggingFace)
6. Stream output → SSE events + mini-terminal
7. Validate output (guardrails)
8. Store result in variables map
9. Save to cache if enabled
10. Persist checkpoint to SQLite

Process management (Claude CLI):

Each step spawns claude --print --output-format stream-json
Timeout: configurable per-step, default 5 minutes
Escalation: SIGTERM → wait 3s → SIGKILL
Heartbeat: every 30s, checks if process is still alive
Cancellation: kills all active processes for an execution

Multi-provider routing:

Steps specify model: "provider/model" or provider: "openrouter"
Auto-detection: claude-* → Claude CLI, gpt-* → OpenAI, etc.
OpenAI-compatible: streaming and non-streaming, with tool calling support

providers.ts — LLM Provider Management

Provider	Type	API Format	Key Required
Claude	`claude`	CLI subprocess	No (CLI auth)
OpenRouter	`openrouter`	OpenAI-compatible	Yes
OpenAI	`openai`	OpenAI native	Yes
Ollama	`ollama`	OpenAI-compatible (local)	No
HuggingFace	`huggingface`	OpenAI-compatible (router.huggingface.co)	Optional
Custom	`custom`	OpenAI-compatible	Yes

API keys encrypted with AES-256-GCM (scrypt key derivation) at rest.

storage.ts — SQLite Persistence

WAL mode for concurrent read/write
Executions table — ID, chain name, status, input, result, timestamps
Steps table — checkpoint per step with output, tokens, duration
Auto-purge — executions older than EXECUTION_MAX_AGE_DAYS (default 7)

queue.ts — Job Queue

SQLite-backed persistent queue
Priority scheduling (1-10, higher = first)
Configurable worker pool (default 5)
Retry with exponential backoff
Stats: queued, running, done, errored, avg wait time

Data Flow

Variable Resolution

Variables flow through the system:

Chain Input: { topic: "AI" }
 ↓
Variables Map: { "input.topic": "AI", "topic": "AI" }
 ↓
Step 1 runs → output = "Research findings..."
 ↓
Variables Map: { "input.topic": "AI", "research": "Research findings..." }
 ↓
Step 2 prompt: "Summarize: {research}" → "Summarize: Research findings..."

Persistence

Executions → SQLite (occ-main.db) — step checkpoints, token usage, timing
Queue → SQLite (occ-queue.db) — persistent job queue
Schedules → JSON (schedules.json) — cron definitions
Cache → SQLite (occ-main.db) — prompt hash → cached output
State → SQLite (occ-state.db) — key-value store for chains
Vectors → SQLite (occ-vector.db) — FTS5 text embeddings
Graph → SQLite (occ-graph.db) — triple store (subject-predicate-object)
LLM Providers → JSON (llm-providers.json) — encrypted API keys
BLOB Sessions → JSON (blobs/) — canvas state per session

Error Recovery

Scenario	Recovery
Step timeout	SIGTERM → 3s → SIGKILL, step marked error
Process crash	Heartbeat detects within 30s, step marked error
Server restart	Orphaned "running" executions marked error on load
Invalid YAML	Rejected at load time with Zod error details
Claude CLI missing	Detected at startup, clear error message
Execution cancelled	All child processes killed, pending steps skipped
Cache corruption	Silent fallback to re-execution
Provider API error	Retries per provider, error propagated to step
SQLite lock	WAL mode prevents contention; queue retries on busy

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Architecture

Architecture

System Overview

Module Breakdown (18 modules)

Core

Infrastructure

Tools

loader.ts — Chain Loading & Validation

executor.ts — Chain Execution Engine

providers.ts — LLM Provider Management

storage.ts — SQLite Persistence

queue.ts — Job Queue

Data Flow

Variable Resolution

Persistence

Error Recovery

See Also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally