TraceBrain Banner

TraceBrain is an open-source infrastructure for trace-driven observability, supervision, and adaptation in autonomous LLM agent systems.

The platform standardizes heterogeneous agent executions into reconstructable trajectories using a unified delta-based OpenTelemetry (OTLP) schema, enabling scalable trace storage, inspection, retrieval, and downstream analysis across diverse agent frameworks.

Beyond passive logging, TraceBrain transforms execution traces into reusable operational assets for runtime governance, uncertainty-aware supervision, retrieval-driven adaptation, and structured downstream training workflows.

• Delta-based OTLP Schema: Stores incremental state transitions instead of cumulative prompts, significantly reducing redundant context storage.
• Framework-Agnostic Integration: Lightweight SDK and converters support heterogeneous agent ecosystems, including LangChain, SmolAgents, and custom multi-agent systems.
• Reconstructable Execution Trajectories: Supports hierarchical trajectory reconstruction for long-horizon reasoning inspection, anomaly analysis, and operational debugging.

• Runtime Escalation: Agents can request human intervention when detecting anomalous or uncertain behaviors during execution.
• Uncertainty-Driven Supervision: An internal LLM-based Trace Evaluator assigns confidence-aware evaluations and selectively escalates low-confidence trajectories for expert review.
• Command Center UI: Interactive interface for trajectory inspection, human validation, trace governance, and feedback management.

• Hybrid Retrieval Engine: Combines semantic vector retrieval (pgvector) and lexical search (tsvector) through Reciprocal Rank Fusion (RRF) for robust operational memory retrieval.
• Operational Memory Retrieval: Enables agents to reuse successful historical trajectories for retrieval-augmented reasoning and in-context adaptation.
• Automated Curriculum Synthesis: A Curator Agent mines clustered failure patterns and synthesizes structured remediation tasks from operational traces.
• Downstream Adaptation Pipelines: Reconstructed traces and synthesized curricula are exportable as standardized training artifacts (e.g., JSONL) for downstream SFT and RL workflows.

System Architecture

• Your AI Agent: Any agent framework. Uses the TraceClient SDK to send data.
• TraceStore API: The central FastAPI server. Ingests, stores, and serves trace data.
• Database: The persistence layer (PostgreSQL or SQLite).
• Admin Panel UI: A React client in web/ that consumes the TraceStore API.

Tech Stack:

• Backend: FastAPI, SQLAlchemy 2.0, Pydantic V2
• Database: PostgreSQL (production), SQLite (development), pgvector (semantic search)
• Frontend: React (Vite + MUI) in web/
• Deployment: Docker Compose
• AI Integration: LibrarianAgent + AI Judge + Curriculum Curator with multi-provider LLM support
• Embeddings: sentence-transformers (local) or OpenAI/Gemini (cloud)

Take a look at the TraceBrain Command Center in action:

🌐 Welcome to the Command Center
The central hub for agentic trace management, featuring a clean, intuitive, and modern interface.
TraceBrain Homepage

Choose one of three installation paths based on your needs. Each option ends with the same user experience: a unified UI + API at http://localhost:8000.

This is the default path for most users. It automatically provisions a production-ready PostgreSQL + pgvector environment. Option 1 uses pre-built images from Docker Hub.

1. Install the CLI

   pip install tracebrain

2. Initialize

   tracebrain init

   This creates a template .env file for API keys and configuration.

   Open the .env file and add your API keys before continuing. If you skip this step, the containers will start but AI features (Librarian, Judge) will fail.

3. Start the platform

   tracebrain up
   # or lightweight profile:
   tracebrain up --slim

Access: http://localhost:8000 (UI + API)

Note: Option 1 uses pre-built images from Docker Hub, so you don't need Node.js or local build tools.

Docker image profiles:

• quyk67uet/tracebrain:latest (Full, ~2.8GB): includes local embedding stack and supports EMBEDDING_PROVIDER=local.
• quyk67uet/tracebrain:slim (Lite, ~400-500MB): faster to pull, optimized for cloud embeddings (EMBEDDING_PROVIDER=openai or gemini).

If you use Docker, you only need pip install tracebrain to get the CLI. You do not install model dependencies on your host machine; they come from the selected image profile (latest or slim).

Best for fast evaluation without Docker.

1. Install the CLI

   pip install tracebrain

2. Initialize

   tracebrain init

3. Create local DB

   tracebrain init-db

   This creates a local SQLite file and prepares tables.

4. Launch

   tracebrain start

Access: http://localhost:8000 (UI + API)

Technical note: the Python backend serves the bundled React build from its internal static directory, so no separate frontend build step is required.

If you run locally without Docker and want to keep the environment light, install the core package first (pip install tracebrain). When you need a specific provider, add only that extra (for example pip install tracebrain[openai]).

For contributors who plan to modify TraceBrain source code.

1. Clone the repository

   git clone https://github.com/ToolBrain/TraceBrain.git
   cd TraceBrain

2. Backend (editable install)

   pip install -e .
   tracebrain start

3. Frontend (HMR)

   cd web
   npm install
   npm run dev

Access:

• Frontend: http://localhost:5173 (Hot Module Replacement)
• API: http://localhost:8000

TraceBrain supports optional extras to minimize dependencies. Install only what you need.

pip install tracebrain
# Optional extras
pip install tracebrain[embeddings-local] # local embeddings
pip install tracebrain[openai] # OpenAI provider
pip install tracebrain[anthropic] # Anthropic provider
pip install tracebrain[huggingface] # Hugging Face provider SDK
pip install tracebrain[all-llms] # OpenAI + Anthropic + Hugging Face

• Trace: A single execution attempt (an "experiment").
• Episode: A logical group of traces (attempts) aimed at solving a single user task.

Traces

• POST /api/v1/traces - Create a new trace
• POST /api/v1/traces/init - Initialize a trace before spans are available
• GET /api/v1/traces - List all traces
• GET /api/v1/traces/{trace_id} - Get trace details
• POST /api/v1/traces/{trace_id}/feedback - Add feedback to a trace
• GET /api/v1/export/traces - Export raw OTLP traces as JSONL (supports status, min_rating, error_type, min_confidence, max_confidence, start_time, end_time)

Episodes

• GET /api/v1/episodes - List all episodes along with their full traces
• GET /api/v1/episodes/summary - List episodes with aggregated metrics
• GET /api/v1/episodes/{episode_id} - Get episode details with trace summaries
• GET /api/v1/episodes/{episode_id}/traces - Get all full traces in an episode

Analytics

• GET /api/v1/stats - Get overall statistics
• GET /api/v1/analytics/tool_usage - Get tool usage analytics

Natural Language Queries

• POST /api/v1/natural_language_query - Query traces with natural language
  • Uses Librarian provider/model from Settings (stored in DB)
  • Requires the matching provider API key in environment ({PROVIDER}_API_KEY)
  • Supports session_id for chat memory and returns suggestions
• GET /api/v1/librarian_sessions/{session_id} - Load stored chat history

AI Evaluation

• POST /api/v1/ai_evaluate/{trace_id} - Evaluate a trace using the configured Judge provider/model
• POST /api/v1/ops/batch_evaluate - Run AI judge over recent traces missing tracebrain.ai_evaluation
• POST /api/v1/traces triggers background evaluation when no AI draft exists

Operations

• DELETE /api/v1/ops/traces/cleanup - Delete traces that match cleanup filters

Semantic Search

• GET /api/v1/traces/search - Find similar traces using vector similarity

Governance Signals

• POST /api/v1/traces/{trace_id}/signal - Update trace status/priority

Curriculum

• POST /api/v1/curriculum/generate - Generate tasks from failed/low-rated traces using configured Curator provider/model
• GET /api/v1/curriculum - List pending curriculum tasks
• GET /api/v1/curriculum/export - Export curriculum tasks as JSONL
• DELETE /api/v1/curriculum/{task_id} - Delete a curriculum task
• DELETE /api/v1/curriculum - Delete all curriculum tasks
• PATCH /api/v1/curriculum/{task_id}/complete - Mark a curriculum task as complete
• PATCH /api/v1/curriculum/complete - Mark all curriculum tasks as complete

History

• GET /api/v1/history - Retrieve history of viewed traces and episodes
• POST /api/v1/history - Add or update last time trace or episode was viewed
• DELETE /api/v1/history - Clear all traces and episodes in viewed history

Settings

• GET /api/v1/settings - Retrieve current LLM routing settings
• POST /api/v1/settings - Update LLM routing + provider API keys (librarian_*, judge_*, curator_*, *_api_key)

Trace status is stored in both the database column status and in attributes.tracebrain.trace.status for UI and API consistency.

Supported statuses:

• running - Trace is in progress or not finalized.
• completed - Trace has been reviewed and finalized.
• needs_review - Trace requires human attention.
• failed - Trace is marked as failed.

A trace is marked as needs_review when one of the following conditions is triggered:

• Runtime Escalation:
  The agent explicitly invokes request_human_intervention during execution.

• Low-Confidence Evaluation:
  tracebrain.ai_evaluation.confidence < 0.75.

• Critical Behavioral Anomalies:
  tracebrain.ai_evaluation.error_type matches one of the following categories: logic_loop, hallucination, invalid_tool_usage, tool_execution_error, format_error, misinterpretation, context_overflow.

• Execution Failure:
  Any span contains otel.status_code = ERROR.

TraceBrain now separates configuration into two layers:

• Runtime routing settings (DB-backed): provider/model for Librarian, Judge, Curator.
• Secrets and infra flags (env): provider API keys, embedding config, debug flags.

Runtime settings are editable from the UI or POST /api/v1/settings, and are persisted in the database. On first startup (when DB settings row does not exist), values are bootstrapped from DEFAULT_* env variables.

For a complete, up-to-date template, copy from .env.example (same content as tracebrain init).

# SQLite (default for development)
DATABASE_URL=sqlite:///./tracebrain_traces.db
# PostgreSQL (for production)
# DATABASE_URL=postgresql://tracebrain:tracebrain_2026_secure@localhost:5432/tracestore
POSTGRES_USER=tracebrain
POSTGRES_PASSWORD=tracebrain_2026_secure
POSTGRES_DB=tracestore
HOST=127.0.0.1
PORT=8000
LOG_LEVEL=info

Use provider-specific key names only:

OPENAI_API_KEY=your_openai_api_key_here
GEMINI_API_KEY=your_gemini_api_key_here
# ANTHROPIC_API_KEY=your_claude_api_key_here
# HUGGINGFACE_API_KEY=your_huggingface_api_key_here

Optional provider base URLs:

# Optional: custom endpoints/proxies
# OPENAI_BASE_URL=https://your-openai-compatible-endpoint/v1
# ANTHROPIC_BASE_URL=https://your-anthropic-endpoint
# HUGGINGFACE_BASE_URL=http://localhost:8000

If TraceBrain runs inside Docker and your provider runs on the host machine, use host.docker.internal instead of localhost in base URLs.

Ollama (OpenAI-compatible) quick setup:

# Local host mode (TraceBrain not in Docker)
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama
# Docker mode (TraceBrain in container, Ollama on host)
# OPENAI_BASE_URL=http://host.docker.internal:11434/v1

Then in TraceBrain Settings:

• Provider: OpenAI
• Model ID: your Ollama model (for example qwen2.5:32b)

TraceBrain uses a resilient OpenAI adapter: it tries /v1/responses first, and automatically falls back to /v1/chat/completions when the endpoint is not supported.

Hugging Face local inference (vLLM/TGI):

If you run a local inference server (vLLM or TGI), set HUGGINGFACE_BASE_URL to your server URL. When this is set, TraceBrain routes Hugging Face traffic to your local endpoint instead of the Hugging Face cloud API.

# Example: local vLLM/TGI endpoint
HUGGINGFACE_BASE_URL=http://localhost:8000
HUGGINGFACE_API_KEY=your_token_if_required

These defaults are used only when settings are not yet stored in DB:

DEFAULT_LIBRARIAN_PROVIDER=openai
DEFAULT_LIBRARIAN_MODEL=gpt-4o-mini
DEFAULT_JUDGE_PROVIDER=gemini
DEFAULT_JUDGE_MODEL=gemini-2.5-flash
DEFAULT_CURATOR_PROVIDER=gemini
DEFAULT_CURATOR_MODEL=gemini-2.5-flash

LIBRARIAN_MODE=api
LLM_DEBUG=false
# Optional for Docker mode
# TRACEBRAIN_IMAGE=quyk67uet/tracebrain:latest
# TRACEBRAIN_IMAGE=quyk67uet/tracebrain:slim
EMBEDDING_PROVIDER=local
EMBEDDING_MODEL=all-MiniLM-L6-v2
# For cloud embeddings
# EMBEDDING_API_KEY=your_embedding_api_key_here
# EMBEDDING_BASE_URL=https://your-embedding-endpoint/v1

Important:

• LLM routing (Librarian/Judge/Curator provider + model) can be changed at runtime from Settings without breaking existing traces.
• Embedding engine (EMBEDDING_PROVIDER + EMBEDDING_MODEL) is infrastructure-level for a database lifecycle.
• Do not switch embedding provider/model on an existing database unless you run a full re-embedding migration. Mixing vector dimensions (for example 384 vs 1536) will break semantic search and experience retrieval.
• For safety, embedding is configured through .env and shown as read-only status in the Settings UI.

GET /api/v1/settings and POST /api/v1/settings use this shape:

{
 "librarian_provider": "openai",
 "librarian_model": "gpt-4o-mini",
 "judge_provider": "gemini",
 "judge_model": "gemini-2.5-flash",
 "curator_provider": "gemini",
 "curator_model": "gemini-2.5-flash",
 "openai_api_key": "sk-...abcd",
 "gemini_api_key": "AIza...wxyz",
 "anthropic_api_key": null,
 "huggingface_api_key": null
}

Notes:

• GET /api/v1/settings returns masked API keys for safety.
• POST /api/v1/settings accepts plain-text API keys when you want to add or rotate keys.
• If a DB key is empty, TraceBrain falls back to the corresponding environment variable (OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY, HUGGINGFACE_API_KEY).
• For OpenAI-compatible local endpoints (Ollama, vLLM, TGI, LM Studio), set OPENAI_BASE_URL and keep provider as OpenAI in Settings.

Example API Usage:

import requests
# Create a trace
response = requests.post("http://localhost:8000/api/v1/traces", json={
 "trace_id": "trace-001",
 "spans": [
 {
 "span_id": "span-001",
 "trace_id": "trace-001",
 "name": "User Request",
 "start_time": "2024-01-01T10:00:00Z",
 "end_time": "2024-01-01T10:00:05Z",
 "attributes": {
 "tracebrain.span.type": "user_request",
 "tracebrain.content.new_content": "What's the stock price of NVIDIA?"
 }
 }
 ]
})
# Add feedback
requests.post("http://localhost:8000/api/v1/traces/trace-001/feedback", json={
 "rating": 5,
 "tags": ["accurate", "fast"],
 "comment": "Great response!",
 "metadata": {
 "outcome": "success",
 "efficiency_score": 0.95
 }
})

The admin UI provides:

• Trace Browser: View all traces with filters
• Trace Details: Expandable span tree visualization and compare related traces
• Feedback Form: Rate and tag traces
• Analytics Dashboard: Stats, tool usage charts
• AI Librarian: Session-aware chat with suggestions and history restore
• AI Evaluation: AI draft is auto-generated and experts verify or edit before finalizing
• Governance Signal: Mark traces with status and priority
• Curriculum: Generate and review training tasks

Frontend dev server (local development only):

cd web
npm install
npm run dev

Semantic search is used in these places:

• API: GET /api/v1/traces/search for vector similarity over traces
• Experience Retrieval: search_similar_traces and search_past_experiences agent tools
• AI Librarian: uses semantic search to surface relevant past traces when enabled

Configure embeddings for vector search and experience retrieval:

# local (default)
EMBEDDING_PROVIDER=local
EMBEDDING_MODEL=all-MiniLM-L6-v2
# cloud (OpenAI/Gemini)
EMBEDDING_PROVIDER=openai
EMBEDDING_API_KEY=your-key
EMBEDDING_MODEL=text-embedding-3-small
# optional for OpenAI-compatible endpoints
EMBEDDING_BASE_URL=https://your-endpoint/v1

When embeddings run: embeddings are created at trace ingest time, not during server startup.

Critical rule: Pick one embedding provider/model per database and keep it stable. Changing embedding dimensions mid-lifecycle can cause pgvector similarity queries to fail.

Do I need local embeddings? No. You can skip embeddings-local entirely and still run the platform. If no embedding provider is configured, traces still ingest and all non-semantic features work normally; only vector search (and features that rely on it) are unavailable.

This section focuses on read/query operations. For logging traces, see the trace_scope section below.

import json
from tracebrain.sdk.client import TraceClient, TraceScope
client = TraceClient(base_url="http://localhost:8000")
# Query traces
traces = client.list_traces()
# Export traces as JSONL
jsonl_payload = client.export_traces(min_rating=4, limit=100)
# Parse JSONL into Python objects
trace_items = [json.loads(line) for line in jsonl_payload.splitlines() if line.strip()]
# Reconstruct messages or turns from OTLP
trace_data = client.get_trace("my-trace-001")
# to_messages: rebuilds chat message list (role/content) from spans
messages = TraceScope.to_messages(trace_data)
# Example: messages[:2] -> [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]
# to_turns: groups messages into conversation turns for UI/analysis
turns = TraceScope.to_turns(trace_data)
# Example: turns[0] -> {"user": "...", "assistant": "..."}
# to_tracebrain_turns: returns TraceBrain-native turn objects with metadata
tracebrain_turns = TraceScope.to_tracebrain_turns(trace_data)
# Example: tracebrain_turns[0] -> {"turn_id": "...", "messages": [...], "span_ids": [...]} 

Use trace_scope for every instrumented agent execution. The scope automatically initializes a trace via /api/v1/traces/init, propagates the trace ID through a context-local runtime store (safe for asynchronous and multi-threaded execution), and uploads the finalized trajectory when execution completes.

This mechanism is required for runtime escalation workflows, ensuring that calls to request_human_intervention are correctly attached to the active execution trace and governance pipeline.

Recommended: use trace_scope (auto init + auto log)

from tracebrain import TraceClient
from my_converters import convert_smolagent_to_otlp
client = TraceClient(base_url="http://localhost:8000")
with client.trace_scope(system_prompt="You are a helpful assistant") as trace:
 agent = MyAgent(system_prompt="You are a helpful assistant")
 agent.run("Summarize this report")
 otlp_trace = convert_smolagent_to_otlp(agent)
 trace["spans"] = otlp_trace.get("spans", [])

Advanced: manual trace ID + manual log

from tracebrain import TraceClient
from tracebrain.sdk.trace_context import set_trace_id, get_trace_id
from my_converters import convert_smolagent_to_otlp
client = TraceClient(base_url="http://localhost:8000")
set_trace_id("trace_123")
agent = MyAgent(system_prompt="You are a helpful assistant")
agent.run("Summarize this report")
otlp_trace = convert_smolagent_to_otlp(agent)
otlp_trace["trace_id"] = get_trace_id() or "trace_123"
client.log_trace(otlp_trace)

When to use:

• Use search_past_experiences to retrieve validated successful trajectories for retrieval-augmented reasoning and operational memory reuse.
• Use search_similar_traces to perform hybrid semantic-lexical retrieval over historical execution traces.
• Use request_human_intervention when the agent encounters uncertain, anomalous, or unresolved execution states requiring expert review.

from tracebrain.sdk import (
 search_past_experiences,
 search_similar_traces,
 request_human_intervention,
)
# Retrieve successful operational trajectories
experiences = search_past_experiences(
 "resolve a tool execution failure",
 min_rating=4,
 limit=3,
)
# Hybrid semantic-lexical retrieval over traces
similar = search_similar_traces(
 semantic_intent="multi-agent coordination failure",
 exact_keywords="LOGIC_LOOP",
 min_rating=4,
 limit=3,
)
# Escalate uncertain behavior for runtime supervision
help_request = request_human_intervention(
 "Execution entered a recurrent reasoning cycle and requires expert validation"
)

TraceBrain uses the TraceBrain OTLP (OpenTelemetry Protocol) format - a delta-based trace schema with parent_id chains for conversation reconstruction.

See docs/Converter.md for:

• OTLP schema explanation (parent_id, new_content, delta-based design)
• Step-by-step conversion recipe
• Python template code with examples

Quick Example:

import uuid
from tracebrain.core.schema import TraceBrainAttributes, SpanType
def convert_my_agent_to_otlp(agent_data):
 spans = []
 parent_id = None
 for step in agent_data.steps:
 spans.append({
 "span_id": str(uuid.uuid4()),
 "parent_id": parent_id, # Chain spans together
 "name": step.action,
 "attributes": {
 TraceBrainAttributes.SPAN_TYPE: SpanType.LLM_INFERENCE,
 TraceBrainAttributes.LLM_NEW_CONTENT: step.output, # Delta content only
 TraceBrainAttributes.TOOL_NAME: step.tool_name,
 }
 })
 parent_id = spans[-1]["span_id"]
 return {"trace_id": agent_data.id, "spans": spans}

TraceBrain/
├── src/
│ ├── tracebrain/ # Core package logic
│ │ ├── api/v1/ # FastAPI REST endpoints
│ │ ├── core/ # TraceStore, schema, agent logic
│ │ ├── db/ # Database session management
│ │ ├── resources/ # Bundled Docker + sample data
│ │ ├── static/ # Bundled React build artifacts
│ │ ├── sdk/ # Client SDK
│ │ ├── cli.py # CLI commands
│ │ └── main.py # FastAPI app entry
├── docs/ # Documentation
├── web/ # React source code (contributors)
├── pyproject.toml # Project metadata
└── README.md

No automated test suite is included yet.

tracebrain seed

No migration tooling is included yet. For schema changes:

1. Update models in src/tracebrain/db/base.py
2. Recreate the database:
   • SQLite (local): delete tracebrain_traces.db, then run tracebrain init-db
   • PostgreSQL (Docker): docker compose -f src/tracebrain/resources/docker/docker-compose.yml down -v then tracebrain up

When querying JSONB fields:

from sqlalchemy import func, cast
from sqlalchemy.dialects.postgresql import JSONB
# Extract text from JSONB
span_type = func.jsonb_extract_path_text(Span.attributes, "tracebrain.span.type")
# Cast for complex queries
rating = func.jsonb_extract_path_text(cast(Trace.feedback, JSONB), "rating")

• Building Your Own Trace Converter - Complete guide for integrating custom agent frameworks
• LLM Provider Guide - Use TraceBrain LLM providers and attach usage metadata
• Trace Reconstruction Guide - Rebuild full context from delta traces for training
• Sample OTLP Traces - Example trace files
• API Documentation - Interactive OpenAPI docs (when server is running)
• Docker Setup Guide - Docker-specific instructions

Contributions are welcome! Here's how to get started:

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and test thoroughly
4. Commit with clear messages: git commit -m 'Add amazing feature'
5. Push to your fork: git push origin feature/amazing-feature
6. Open a Pull Request

Development Guidelines:

• Follow PEP 8 style guide
• Add tests for new features
• Update documentation as needed
• Ensure Docker builds pass

If code changes aren't picked up after tracebrain up --build:

tracebrain down
docker compose -f src/tracebrain/resources/docker/docker-compose.yml build --no-cache
tracebrain up

Ensure PostgreSQL is running and check connection string in src/tracebrain/config.py:

DATABASE_URL = "postgresql://traceuser:tracepass@localhost:5432/tracedb"

After updating store.py, rebuild Docker containers to apply JSONB query fixes.

This project is licensed under the MIT License - see the LICENSE file for details.

• Built with FastAPI
• Database powered by SQLAlchemy
• UI with React (Vite) + MUI
• Inspired by OpenTelemetry standards

Made with ❤️ for the AI agent community

For questions or support, please open an issue on GitHub.