A local-first Retrieval-Augmented Generation (RAG) assistant for industrial vision-systems documentation (Pekat, Mechmind, Zivid, LMI, and others). Ingest PDFs and HTML help articles into Qdrant, query them through a streaming chat UI, and optionally fall back to web search when local context is insufficient.

• Chat UI — Dark-theme SPA with SSE streaming, conversation history, stop control, thumbs up/down feedback, optional dev analytics, and a documents panel (upload, ingest progress, delete).
• Vendor/product scoping — Toolbar dropdowns populated from GET /vendors; keyword inference from the question when filters are unset; explicit filters always win.
• RAG pipeline — Query condensation for follow-ups, HyDE, parallel QueryOrchestrator, cross-encoder reranking, semantic cache, sufficiency check, and SearXNG web fallback.
• Ingest v2 — PDF + HTML (URL list or single URL), schema v2 Qdrant payloads, ingest manifest, collision-proof filenames, pending caption/video sidecars.
• Provider abstraction — Ollama (default LLM + embed), local GPU or TEI embeddings, OpenAI-compatible chat endpoints (vllm / tgi / openai_compatible), optional VLM; Qdrant local, sharded, or cloud.
• Ops tooling — python -m scripts.ingest.reingest_all (full-corpus re-ingest), python -m scripts.ingest.extract_pdf_images + python -m scripts.ingest.caption_worker (PDF/HTML image caption pipeline), python -m scripts.ops.audit, python -m scripts.ops.cleanup, eval suite with CI baseline checks.
• Observability — Langfuse v3 per-query analytics; optional OpenTelemetry spans.

┌──────────────┐ ┌──────────────┐ ┌─────────────────────────────┐
│ Chat UI │────▶│ FastAPI │────▶│ providers/rag_pipeline.py │
│ (SSE) │ │ api/main.py │ │ plan → stream → analytics │
└──────────────┘ └──────────────┘ └──────────────┬──────────────┘
 │
 ┌───────────────────────────────┼───────────────────────────────┐
 ▼ ▼ ▼
 QueryOrchestrator GenerationPlan Langfuse
 (embed ∥ HyDE ∥ cache) (prompt + citations) + OTel
 │
 ┌───────────────┼───────────────┬───────────────┬───────────────┐
 ▼ ▼ ▼ ▼ ▼
 Qdrant Reranker Redis cache SearXNG Ollama / vLLM
 vectors (CrossEncoder) (RedisVL opt.) web search LLM + embed

1. Optional condensation — When chat history is non-empty, the LLM rewrites the follow-up into a standalone retrieval query (original question still goes to the generation prompt).
2. Retrieval scope — Explicit vendor / product from the UI or API; else single-vendor keyword detection on the (condensed) question; multiple vendor keywords → no filter (comparison queries). Zero filtered hits → unfiltered fallback (logged).
3. Retrieval — QueryOrchestrator: embed question (parallel with HyDE when enabled), semantic cache lookup (skipped when history or filters are active), Qdrant search with payload filters, optional hybrid image retrieval, rerank.
4. Generation — Sufficiency check and/or early web fallback; assemble prompt with history + chunk headers; stream LLM tokens; retry with web if answer is insufficient.

PDF / HTML / TXT → read/split (producer thread) → chunk queue
 ↓
 embed batches (worker thread)
 ↓
 Qdrant upsert + doc_registry (main thread)

• HTML — BeautifulSoup extraction; Confluence pages prefer .ak-renderer-document when present (avoids SPA chrome). Heading-aware sections, image/video sidecars (pending_captions.json, pending_videos.json).
• Incremental — data/ingest_manifest.json tracks SHA-256 per source; unchanged files skip re-embed unless --force.
• Metadata — Upload modal and CLI flags set vendor, product, doc type, version, URL.

Note: Deep design notes and agent context live in local-only ARCHITECTURE.md / CLAUDE.md (gitignored).

Each Qdrant point payload includes:

Citations returned to the UI include source, vendor, page, section, and scores when available.

Filter precedence is logged as filter_mechanism: explicit, keyword, or none (Langfuse + dev analytics).

rag-agent/
├── api/main.py # FastAPI: /query, /upload, /vendors, /documents, sessions
├── ui/index.html # Chat UI (vanilla JS, no build step)
├── providers/rag_pipeline.py # RAG query pipeline
├── scripts/ingest/ingest.py # Ingest v2 (PDF, HTML, URL, VTT modes)
├── config/
│ ├── vendors.json # Vendor folders, YouTube channel aliases
│ └── searxng/settings.yml # SearXNG instance config
├── providers/ # LLM, embed, Qdrant, cache, reranker, orchestrator, ...
├── scripts/
│ ├── ingest/ # ingest, reingest, caption_worker, ingest_video, ...
│ ├── ops/ # audit, cleanup, migrate_*, backfill_doc_registry
│ ├── data/ # download_docs, fetch_lmi_urls
│ └── backfill_manifest.py # Manifest backfill for legacy Qdrant sources
├── eval/ # Retrieval/answer eval, CI baseline
├── tests/ # Unit tests (see tests/TESTING.md)
├── data/ # Runtime: uploads, manifest, registries (gitignored)
├── logs/ # Operational logs (gitignored)
├── docs/ # Vendor manuals + internal notes (content gitignored)
├── docker-compose.yml # qdrant, redis, searxng, api, langfuse stack
└── Dockerfile # API container image

• Python 3.11+ with a virtual environment
• Ollama on the host (ollama serve) for default LLM and embeddings
• Docker for Qdrant, Redis, SearXNG, and optional Langfuse

ollama pull qwen3.5:9b
ollama pull qwen3-embedding:0.6b
ollama pull llava:7b

Default stack: LLM qwen3.5:9b, embed qwen3-embedding:0.6b (1024-dim), vision llava:7b for image captions. Set OLLAMA_THINK_ENABLED=false for qwen3.x RAG (see .env.example).

git clone https://github.com/charbelfakh/rag-agent.git
cd rag-agent
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
copy .env.example .env

Copy .env.example to .env. Common variables:

See .env.example for the full list (orchestrator flags, blob storage, sharding, multimodal, etc.).

docker compose up -d qdrant redis searxng
ollama serve
python api/main.py

Open http://localhost:8000 . Hard-refresh after UI changes.

# PDF
python -m scripts.ingest.ingest data/pekat/manual.pdf --vendor pekat --doc-type manual
# HTML from URL list
python -m scripts.ingest.ingest --url-list data/lmi/urls_kb.txt --vendor lmi --doc-type article --product gocator
# UI: Documents panel → upload with metadata modal

Resumable pipeline: text ingest → PDF image extraction → vision caption drain → report.

# Dry-run first
python -m scripts.ingest.reingest_all --vendors lmi pekat --dry-run
# Full run (GPU-heavy during caption phase — avoid concurrent chat queries)
python -m scripts.ingest.reingest_all --vendors lmi pekat
# Or run stages individually:
python -m scripts.ingest.extract_pdf_images --all
python -m scripts.ingest.caption_worker --vendor lmi --reconcile-missing -v
python -m scripts.ingest.caption_worker --vendor pekat -v

State is saved to data/reingest_state.json; reports go to reingest_report_*.md (both gitignored).

python -m scripts.data.download_docs --list
python -m scripts.data.download_docs --vendor pekat

python -m scripts.ops.audit --detail --vendor lmi
python -m scripts.ops.audit --json logs/audits/report.json
python -m scripts.ops.cleanup --dry-run --delete-sources data/lmi/sources_to_delete.txt
python -m scripts.ops.cleanup --retag data/lmi/sources_to_retag.txt --product hexsight --yes

When API_KEY is set, send X-API-Key on protected routes.

{
 "question": "How does surface flatness work?",
 "top_k": 10,
 "history": [],
 "vendor": "pekat",
 "product": null
}

Omit vendor and product when using "All vendors" / "All products" in the UI.

Enable Langfuse:

ANALYTICS_ENABLED=true
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=http://localhost:3000

docker compose up -d langfuse-web langfuse-worker langfuse-postgres langfuse-clickhouse langfuse-minio langfuse-redis

Dev analytics in the UI: Settings → Show query analytics.

python eval/run_retrieval_eval.py
python eval/run_retrieval_eval.py --write-baseline
python eval/run_retrieval_eval.py --check-baseline
python eval/run_answer_eval.py
# Caption ablation (text-only vs full corpus)
python eval/run_retrieval_eval.py --dataset eval/dataset_caption.jsonl --content-type-filter text
python eval/run_retrieval_eval.py --dataset eval/dataset_caption.jsonl --content-type-filter none

CI runs unit tests and baseline checks via .github/workflows/eval.yml.

pip install -r requirements-dev.txt
python -m pytest tests/ -q

339 tests across sprints A–Q, API endpoints, ingest v2, caption worker, and coverage audit. Requires pytest-asyncio (included in requirements-dev.txt). Policy: new modules, env knobs, and API routes need tests in the same change (tests/TESTING.md).

• Cross-encoder cache for repeated retrieval patterns
• Automated golden-set expansion from production sampling (eval/production_sampler.py)
• Multi-tenant API keys and per-tenant collections
• Hosted deployment guide (Qdrant Cloud + GPU embed sidecar)

Enable via .env — each has tests in tests/test_sprint_*.py:

# Retrieval
QUERY_ORCHESTRATOR_ENABLED=true
TWO_STAGE_RETRIEVAL_ENABLED=true
HYBRID_RETRIEVAL_ENABLED=true
MAP_REDUCE_RETRIEVAL_ENABLED=true
SPECULATIVE_GENERATION_ENABLED=true
# Ingest / storage
INGEST_QUEUE_ENABLED=true
BLOB_STORAGE_ENABLED=true
MULTIMODAL_IMAGE_INGEST_ENABLED=true
QDRANT_VENDOR_SHARDING=true
# Cache
SEMANTIC_CACHE_BACKEND=redisvl
# Infra
OTEL_ENABLED=true
SESSION_STORAGE_ENABLED=true

See .env.example for defaults and Docker Compose profiles (workers, gpu-embed, vllm-pool, multimodal).

• Ollama on host — Not in Docker; [WinError 10061] means Ollama is not running on port 11434.
• Client-side chat history — Conversations default to browser localStorage unless session storage is enabled.
• Semantic cache — Skipped when chat history or vendor/product filters are active.
• Large ingests — Full vendor corpora (e.g. 270 LMI articles) take significant time on local Ollama embed.
• Web fallback — Quality depends on SearXNG instance and upstream engines; not vendor-scoped.
• Corpus gaps — Strong retrieval cannot answer topics absent from ingested docs. Some Pekat KB3150 PDFs are print-to-PDF shells (text-only, no figures); captioning cannot recover missing screenshots.
• GPU contention — On 8 GB cards, do not run python -m scripts.ingest.caption_worker concurrently with live chat queries (vision + LLM + embed compete for VRAM).

See LICENSE.