raven-retrieval

This is a retrieval research framework for comparing retrieval methods head-to-head on equal footing — same datasets, same chunking, same metrics, same statistical tests. It implements 19 pipelines and benchmarks them all against each other with proper significance testing.

The core question it answers: does applying ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval quality over simpler approaches? Everything else is scaffolding to answer that question properly.

These aren't standalone pipelines — they wrap other retrievers to add capabilities.

Everything starts with the encoder. Two options:

SBERT (for dense/hybrid baselines): all-MiniLM-L6-v2, produces a single 384-dim vector per chunk. Fast, trained on millions of retrieval pairs, works out of the box.

ColBERT (for late interaction): bert-base-uncased backbone → Linear(768 → 128) projection → L2 normalize per token. Produces a matrix of per-token embeddings instead of a single vector. The projection head is Xavier-initialized but NOT trained on retrieval triples in the default setup. This is a known limitation — the ColBERT encoder needs fine-tuning to reach its potential. The code supports training via ColbertContrastiveEncoder with contrastive loss or InfoNCE.

The encoder lives in src/encoder/colbert_encoder.py. Key methods:

• encode_query(text, max_length=64) → tensor of shape (n_query_tokens, 128)
• encode_document(text, max_length=256) → tensor of shape (n_doc_tokens, 128)
• encode_pooled(texts, strategy="mean"|"max"|"cls") → single vector per text (for clustering/GMM)
• get_temperature() → learned temperature parameter (exp(log_tau))

ColBERT's scoring function. For query Q = [q1, ..., qm] and document D = [d1, ..., dn]:

MaxSim(Q, D) = Σi maxj sim(qi, dj)

For each query token, find the most similar document token (cosine similarity), then sum those maximums across all query tokens. This captures fine-grained token-level relevance that single-vector cosine misses.

Implemented in src/maxsim/brute_force.py — works entirely in numpy, no torch required for scoring. The maxsim_score_batch() function handles concatenated document embeddings with offset tracking for efficiency.

The hierarchical tree is built offline (one-time cost):

1. Chunk the corpus into ~100-token segments (Level 0 leaf nodes)
2. Embed each chunk (SBERT for pooled embeddings, ColBERT for token embeddings)
3. Cluster using two-step UMAP + GMM:
   • Global step: n_neighbors ≈ sqrt(N-1) preserves corpus-wide structure
   • Local step: n_neighbors ≈ 10 within each global cluster captures fine-grained structure
   • Cluster count selected via BIC (Bayesian Information Criterion) to penalize overfitting
   • Soft assignment: nodes with P(cluster) > 0.1 go to multiple clusters
4. Summarize each cluster (BART abstractive, or extractive TF-IDF fallback if model fails)
5. Embed the summaries, add as parent nodes
6. Repeat steps 3-5 until clusters have ≤ min_cluster_size members (default: 3)

The tree ends up with levels of increasing abstraction — raw text at the bottom, meta-summaries at the top. Every node stores both pooled embeddings (for single-vector retrieval) AND per-token ColBERT embeddings (for late interaction).

Built in src/raptor/builder.py. The LateInteractionRaptor in src/combined/late_raptor.py does the same thing but specifically stores ColBERT token embeddings at every node.

Collapsed: Flatten all nodes (all levels) into a single pool. MaxSim query against every node. Take top-k. Simple, finds the best matching node regardless of level.

Traversal: Start at root nodes. MaxSim score each, take top-k. Descend to their children. Repeat until you hit leaf nodes. More efficient, naturally moves from abstract to specific.

For when brute-force over millions of documents is too slow:

1. K-means all token embeddings into centroids (FAISS)
2. Encode each document as centroid IDs + residuals
3. Query: find top centroids closest to query tokens → candidate documents via centroid overlap → decompress residuals → full MaxSim reranking

Implemented in src/maxsim/approximate.py. The ApproximateMaxSim class also has compute_fidelity() to measure how much quality you lose vs brute-force.

Standard ColBERT stores 128 floats per token = 512 bytes. ColBERTv2 compresses this:

1. Learn centroids (K-means)
2. For each token: store centroid ID (1 byte) + quantized residual (2 bits per dim)
3. Total: ~17 bytes per token instead of 512. That's ~30x compression.

At query time: centroid interaction for candidate generation, decompress residuals for full MaxSim. Implemented in src/maxsim/compression.py with ResidualCompressor and CompressedCorpusIndex.

Instead of dense vectors, SPLADE produces sparse vectors over the BERT vocabulary (~30k dimensions). The trick: use BERT's MLM (Masked Language Model) head, apply ReLU + log(1+x) activation, take max across sequence positions.

This naturally expands terms — "cat" might activate "feline", "kitten", "pet" in the vocabulary. The result is a sparse vector that combines lexical matching (like BM25) with semantic expansion (like dense retrieval).

Implemented in src/baselines/splade.py. The get_expansion_terms() method shows you exactly which terms SPLADE would add for a given input — useful for interpretability.

Instead of embedding the query directly:

1. Generate a hypothetical answer to the query (using TinyLlama or a template fallback)
2. Embed the hypothetical answer
3. Retrieve real documents similar to the hypothetical

The insight: a hypothetical answer is semantically closer to real documents than the raw question. The embedding model's "dense bottleneck" filters out hallucinated details.

Implemented in src/baselines/hyde.py. Falls back to 3 rotating templates based on query hash if no LLM is available.

When you chunk a document, each chunk loses its document-level context. "The company's revenue grew by 3%" — which company? What quarter?

Solution: prepend a short context prefix to each chunk before embedding. Something like "Document: ACME Corp Q2 2023 SEC filing. The company's revenue grew by 3%..."

Three flavors in src/baselines/contextual.py:

• ContextualDenseRetriever: embed context-enriched chunks
• ContextualBM25Retriever: BM25 over context-enriched chunks
• ContextualHybridRetriever: both + RRF

Anthropic reported 35-49% reduction in failed retrievals with this technique.

Standard pipeline: chunk text → embed each chunk independently. Problem: each chunk's embedding has no awareness of surrounding context.

Late chunking flips this: embed the entire document first → then split the resulting token embeddings by position → pool per chunk.

The key: when the transformer processes the full document, each token's embedding is contextualized by ALL other tokens, including those in other chunks. So chunk embeddings naturally preserve cross-chunk context.

Implemented in src/baselines/late_chunking.py. Limited by BERT's 512-token context window in the default setup — needs a long-context model (like Jina-embeddings-v2 with 8K tokens) to really shine.

Query Decomposition (QueryDecomposer in src/baselines/agentic.py):

• Detects multi-hop queries via conjunctions ("and", "but also"), comparison patterns ("compare", "vs"), and relative clauses ("that", "which")
• Splits into sub-queries, retrieves for each, aggregates scores
• Also supports LLM-based decomposition (optional)

Reflection (ReflectionRetriever):

• After initial retrieval, evaluates context sufficiency using keyword coverage
• If <50% of query keywords appear in retrieved context, reformulates and re-retrieves
• Up to 2 iterations, each reformulation gets more specific or broader

Multi-Hop (MultiHopRetriever):

• Combines decomposition + per-sub-query retrieval + score aggregation
• Documents appearing in results for multiple sub-queries get higher aggregate scores

This is where the framework takes itself seriously.

All computed via BEIR's EvaluateRetrieval wrapping pytrec_eval (the standard IR evaluation tool):

• nDCG@k: Normalized Discounted Cumulative Gain at k ∈ {1, 3, 5, 10, 100}
• MAP@k: Mean Average Precision
• Recall@k: Fraction of relevant documents retrieved
• Precision@k: Fraction of retrieved documents that are relevant

Not just "pipeline A scored higher than pipeline B." Actual statistical tests:

• Paired bootstrap test: 10,000 resamples per comparison. Computes observed difference, p-value, and 95% confidence interval
• Paired t-test: Cross-check with parametric test (scipy ttest_rel)
• Bonferroni correction: Adjusts significance threshold for multiple comparisons (6 pairwise tests across 4 pipelines → α/6)

All implemented in src/eval/significance.py. The run_all_pairwise_tests() function handles all C(n,2) pairs automatically.

HTML Dashboard (src/eval/visualize.py): Self-contained HTML file with embedded SVG charts:

• Bar chart: nDCG@10 by pipeline
• Grouped bar chart: nDCG@k breakdown
• Radar chart: multi-metric comparison
• Scatter plot: latency vs quality tradeoff
• Significance table with color-coded results

Markdown Report (src/eval/report.py): Auto-generated with:

• nDCG table (best highlighted in bold)
• Recall table
• Latency table with relative speedup
• Statistical significance table
• Key findings analysis
• Production and research recommendations

Expectations are written before the benchmark run (see experiments/preregistration/template.md). Results are compared against stated expectations after the run.

Core Tests (no torch required):

==================================================
Raven-Retrieval Core Test Suite
==================================================
 ✅ chunker_basic
 ✅ chunker_overlap
 ✅ chunker_corpus
 ✅ tree_ops
 ✅ maxsim_basic
 ✅ maxsim_ranking
 ✅ maxsim_batch
 ✅ umap_reduction
 ✅ cluster_count
 ✅ soft_assignment
 ✅ extractive_summarizer
 ✅ extractive_empty
 ✅ compression_roundtrip
 ✅ compression_ratio
 ✅ compression_save_load
 ✅ bootstrap_known_diff
 ✅ bootstrap_no_diff
 ✅ bonferroni
 ✅ pairwise
 ✅ dashboard
 ✅ report
 ✅ query_decomposition
 ✅ centroid_index
Results: 42 passed, 0 failed, 42 total assertions
🎉 ALL TESTS PASSED

Full pytest Suite (torch required):

55 passed, 2 warnings in 85.48s
Tests by module:
 test_integration.py 3/3 passed (pipeline wiring, tree flat retrieval, MaxSim end-to-end)
 test_maxsim.py 6/6 passed (scoring, ranking, batch, centroid index, approximate, fidelity)
 test_metrics.py 9/9 passed (per-query nDCG, recall, precision, MAP, trec_eval alignment)
 test_pipelines_smoke.py 11/11 passed (DI-based contract tests for every retriever)
 test_raptor.py 6/6 passed (chunker, UMAP, cluster count, soft cluster, tree ops)
 test_significance.py 5/5 passed (bootstrap, t-test, Bonferroni, pairwise)
 test_utils.py 15/15 passed (chunking, aggregation, RRF, timer, masking, L2 normalize)

Run ID: enhanced_scifact_1784635462 Date: 2026年07月21日 Hardware: Google Colab T4 GPU (15.6 GB VRAM) Python: 3.12 | PyTorch: 2.11.0+cu128 | CUDA: 12.8 Total runtime: ~45 minutes (all 7 cells) Seed: 42 (numpy + torch)

Run via 7 Colab cells: Cell 2 (baselines), Cell 3 (heavy), Cell 4 (RAPTOR), Cell 5 (agentic+graph), Cell 6 (HotpotQA), Cell 7 (results), Cell 8 (ColBERT training, optional). See BENCHMARK_RESULTS.md for the full cell-by-cell breakdown.

*SciFact (small): 10 queries — perfect 1.0 is small-sample luck. The 100-query main run shows HyDE at 0.7119. Repeat run confirms reproducibility (identical nDCG with seed=42).

Key finding: The ranking flips between datasets. HyDE wins SciFact (single-hop), Hybrid RAG wins HotpotQA (multi-hop). No single pipeline dominates across both tasks.

Pipelines 6–9 (Heavy): Cell 3 timed out. Pipelines 10–12 (RAPTOR): Cell 4 hit UMAP bug (fixed, re-run needed). Pipelines 13–16 (Agentic): 2/4 completed. See BENCHMARK_RESULTS.md for details.

With 100 queries (vs 10 in the earlier run), we get much more reliable averages. The earlier 10-query run showed HyDE at a perfect 1.0 — that was a small-sample artifact. The real picture on100 queries is more nuanced and more honest.

Key findings: BM25 PRF is significantly worse than everything else (p ≈ 0.000, survives Bonferroni). Top 4 pipelines are statistically indistinguishable. With 100 queries, we have real statistical power — the v0.2 "zero variance" bug is fixed.

HyDE wins, but not by much (nDCG@10 = 0.712). The earlier10-query run showed HyDE at a perfect 1.0 — that was small-sample luck. On100 queries, HyDE still comes out on top, but only ~2 points above Naive Dense. The hypothetical document approach genuinely helps bridge the query-document semantic gap, even with simple templates instead of an LLM.

Naive Dense RAG is surprisingly competitive (nDCG@10 = 0.696). Don't sleep on the basics. SBERT with cosine similarity on well-chunked text is hard to beat for single-hop scientific claims. It's also the second-fastest to query.

Contextual Hybrid ≈ Hybrid ≈ Dense. All three are within ~3 points of each other and statistically indistinguishable. Adding Anthropic-style context prefixes to chunks didn't move the needle on SciFact — scientific abstracts are already self-contained, so the "lost in the middle" problem these techniques address just doesn't apply here.

BM25 + Rocchio PRF is significantly worse (nDCG@10 = 0.529). This is the only pipeline with real statistical evidence against it — p≈0.000 against every other method, surviving Bonferroni correction. TF-IDF query expansion actively hurts on precise scientific terminology. The expansion terms add noise, not signal.

Latency is a different conversation. BM25 PRF indexes in1.2 seconds (no neural encoding). The neural pipelines take25-28 seconds to index. If you're building a system where speed matters more than accuracy, BM25 is still valid — just know what you're giving up.

The ranking flips! HyDE wins SciFact (single-hop), Hybrid RAG wins HotpotQA (multi-hop). BM25+Dense fusion catches different reasoning hops that a single semantic similarity misses. Contextual Hybrid benefits more from context prefixes on multi-hop queries.

Significant (Bonferroni): Hybrid vs BM25 PRF (p=0.0045), BM25 PRF vs Contextual (p=0.0031). All other pairs: not significant.

Note: All scores are high (0.87–0.92) because the 2,000-doc subsample makes retrieval easier than the full 5.2M HotpotQA corpus. RAPTOR + Late Interaction results pending (Cell 4 re-run with fixed clustering).

v0.2 significance tests were invalid (per-query scores all zeros, BEIR API bug). Numbers are consistent with v0.3 Colab run.

1. Encoder not trained on retrieval triples . The ColBERT projection head is Xavier-initialized by default, which hurts late interaction quality vs fully-trained SBERT. Fix: make train-colbert (add --hard-negatives for BM25-mined hard negatives), then --colbert-checkpoint checkpoints/final_model.pt to the benchmark runner. This is the single biggest quality lever for late interaction.

2. Soft-clustering ≈ hard assignment on short docs. With short chunks (~100 tokens), the GMM soft assignments tend to converge to near-hard assignments. Measured via compute_soft_assignment_rate(). Matches the Stanford CS224N RAPTOR reproduction.

3. RAPTOR summarizer may use extractive fallback. If BART fails to load (transformers version issues, memory constraints), falls back to TF-IDF extractive summarization. A _load_failed flag prevents the infinite-retry bug; the fallback is logged so tree quality is properly attributable.

4. SPLADE is slow on CPU. Each chunk requires a BERT MLM forward pass (now batched). Use GPU, or accept that indexing takes longer than SBERT.

5. Late Chunking limited by context window. Standard BERT caps at 512 tokens, so the "full document" is actually truncated. Needs a long-context embedding model (Jina-embeddings-v2, etc.) to realize the full benefit.

6. HotpotQA needs >6GB RAM . The 5.2M-document corpus exceeds 6GB when encoded with SBERT. Fix: --max-docs 2000 subsamples the corpus while always preserving judged documents so every metric stays valid. The unified runner also reuses one SBERT/ColBERT/BART instance across all pipelines (critical on a 4-8GB machine).

7. Per-query significance requires the numpy implementation. BEIR's EvaluateRetrieval.evaluate() returns corpus-averaged floats, not a per-query dict — using it for per-query scores (the v0.2 bug) silently produces all-zero arrays and meaningless p-values. v0.3 computes per-query nDCG in pure numpy (trec_eval-compatible), pinned by tests/test_metrics.py.

raven-retrieval/
├── .github/workflows/
│ └── tests.yml # 3-job CI: core tests (3.10/3.11/3.12 matrix) → full pytest → fast benchmark
├── configs/
│ ├── scifact.yaml # 5 pipelines, SciFact defaults
│ ├── hotpotqa.yaml # 4 pipelines, HotpotQA defaults
│ └── full_ablation.yaml # Full registry, comprehensive
├── experiments/
│ └── preregistration/
│ └── template.md # Pre-registered hypotheses template
├── src/
│ ├── __init__.py
│ ├── utils.py # SHARED: chunking, doc-score aggregation, RRF, timing, masking (kills 8x duplication)
│ ├── config.py # Central config: 7 nested dataclasses, YAML/JSON, dot-notation overrides, --max-docs/--colbert-checkpoint
│ ├── encoder/
│ │ ├── __init__.py
│ │ └── colbert_encoder.py # ColbertEncoder + ColbertContrastiveEncoder (contrastive/InfoNCE) + batched masked encoding + checkpoint I/O
│ ├── maxsim/
│ │ ├── __init__.py
│ │ ├── brute_force.py # maxsim_score, brute_force_rank, maxsim_score_batch, brute_force_rank_fast (1 matmul), pack_doc_embeddings
│ │ ├── approximate.py # CentroidIndex (FAISS K-means) + ApproximateMaxSim (PLAID-style, vectorized)
│ │ └── compression.py # ResidualCompressor (30x, vectorized) + CompressedCorpusIndex
│ ├── raptor/
│ │ ├── __init__.py
│ │ ├── chunker.py # TextChunker: fixed-size word chunks with overlap
│ │ ├── clustering.py # Two-step UMAP + GMM: global_local_cluster, soft_cluster, BIC selection
│ │ ├── summarizer.py # LLMSummarizer (BART/T5 + extractive fallback, no infinite retry) + ExtractiveSummarizer (TF-IDF)
│ │ ├── tree.py # TreeNode (token embeddings + pooled) + RaptorTree (traverse/collapse)
│ │ └── builder.py # RaptorBuilder: chunk → embed → cluster → summarize → repeat (accepts shared model/summarizer)
│ ├── combined/
│ │ ├── __init__.py
│ │ └── late_raptor.py # LateInteractionRaptor: novel RAPTOR + ColBERT MaxSim at every node (batched, mask-trimmed, fix)
│ ├── baselines/
│ │ ├── __init__.py # Lazy imports via __getattr__ (avoids pulling torch when not needed)
│ │ ├── dense.py # DenseRetriever: SBERT → cosine, aggregate by doc (model injection for tests)
│ │ ├── hybrid.py # HybridRetriever: BM25 + Dense + RRF
│ │ ├── hyde.py # HyDERetriever: TinyLlama (actually loads now) or template fallback → embed hypothetical → retrieve
│ │ ├── splade.py # SPLADERetriever (MLM logits → sparse, inverted index, batched) + HybridSPLADERetriever
│ │ ├── bm25_prf.py # BM25PRFRetriever: two-stage Rocchio PRF (weighted rerank of candidates only)
│ │ ├── contextual.py # ContextualChunker + Contextual(Dense|BM25|Hybrid)Retriever
│ │ ├── late_chunking.py # LateChunkingEncoder (embed full doc → split → pool) + LateChunkingRetriever
│ │ ├── agentic.py # QueryDecomposer + ReflectionRetriever (real text_lookup!) + MultiHopRetriever
│ │ ├── reranker.py # CrossEncoderReranker (ms-marco-MiniLM) + TwoStageRetriever
│ │ └── graph_retrieval.py # DocumentGraph (cosine edges + label propagation, memory-safe blocked) + GraphRetriever
│ ├── eval/
│ │ ├── __init__.py
│ │ ├── datasets.py # BEIR download/load/subsample + subsample_corpus (judged-doc-preserving)
│ │ ├── metrics.py # run_beir_evaluation + REAL per-query nDCG/recall/precision/map (numpy, trec_eval-compatible)
│ │ ├── significance.py # paired_bootstrap_test, paired_t_test, bonferroni_correction, run_all_pairwise_tests
│ │ ├── ablation.py # AblationRunner + collect_per_query_scores (now returns real values)
│ │ ├── visualize.py # SVG charts (bar, grouped bar, radar, scatter) + generate_dashboard (HTML)
│ │ └── report.py # generate_report (Markdown) with CLI entry point
│ ├── training/
│ │ └── train_colbert.py # generate_triples_from_beir (--hard-negatives via BM25) + train_colbert_encoder (InfoNCE)
│ └── validation/
│ └── cross_check.py # ReferenceCrossValidator: our ColBERT vs ColBERTv2 (ragatouille), Spearman correlation
├── tests/
│ ├── run_core_tests.py # Core suite: numpy-only, runs in CI without torch
│ ├── test_utils.py # Shared utilities (chunking, aggregation, RRF, timer, masking)
│ ├── test_metrics.py # Per-query nDCG/recall/precision/map pinned against trec_eval semantics
│ ├── test_pipelines_smoke.py # DI-based contract tests for every retriever (run without torch/transformers)
│ ├── test_maxsim.py # scoring, ranking, batch, centroid index, approximate, fidelity (faiss-guarded)
│ ├── test_raptor.py # chunker, UMAP, cluster count, soft cluster, tree ops, assignment rate (umap-guarded)
│ ├── test_significance.py # bootstrap known/no diff, t-test agreement, Bonferroni, pairwise
│ └── test_integration.py # pipeline wiring, tree flat retrieval, MaxSim end-to-end
├── run_enhanced_benchmark.py # Unified runner: 19-pipeline registry, shared models, separate index/query timing, error capture
├── Makefile # make test, test-core, benchmark, benchmark-fast, benchmark-baselines, benchmark-trained, train-colbert, report, clean
├── setup.py # pip install raven-retrieval[full,dev] with entry points (v0.3.0)
├── requirements.txt # torch, transformers, sentence-transformers, faiss-cpu, rank-bm25, umap-learn, scikit-learn, beir, scipy, numpy, tqdm
├── requirements-dev.txt # pytest, flake8, mypy
├── requirements-validation.txt # + ragatouille (for reference ColBERT cross-check)
├── README.md # This file
├── METHODOLOGY.md # Research paper methodology: hypotheses, architecture, baselines, evaluation protocol
├── RESEARCH_NOTES.md # Deep research on 11 topics: ColBERTv2/PLAID, SPLATE, MUVERA, RAPTOR improvements, SPLADE, HyDE, GraphRAG, Agentic RAG, Contextual Retrieval, Late Chunking, Token Pooling
├── BENCHMARK_RESULTS.md # Results + analysis + what went wrong + next steps (revised in v0.3)
└── CONTRIBUTING.md # How to add a new pipeline, code style, commit messages

git clone https://github.com/subhansh-dev/raven-retrieval.git
cd raven-retrieval
# Core only (no torch, runs anywhere)
pip install numpy scipy scikit-learn rank-bm25 umap-learn tqdm
# Full (all pipelines)
pip install -r requirements.txt
# Dev (testing, linting)
pip install -r requirements.txt -r requirements-dev.txt
# Or install as a package
pip install -e ".[full,dev]"

# Core tests — no torch required, runs in seconds
make test-core
# or: python tests/run_core_tests.py
# Full test suite (needs torch)
make test
# or: python -m pytest tests/ -v

# Fast benchmark (skip ColBERT/SPLADE/RAPTOR — good for CI or quick checks)
make benchmark-fast
# or: python run_enhanced_benchmark.py --dataset scifact --max-queries 100 --skip-heavy
# Baselines only (dense, hybrid, BM25+PRF, contextual, hyde)
make benchmark-baselines
# Full benchmark (all default pipelines)
make benchmark
# or: python run_enhanced_benchmark.py --dataset scifact --max-queries 100
# Specific pipelines only (from the 19-pipeline registry)
python run_enhanced_benchmark.py --pipelines naive_dense hybrid_rag hyde bm25_prf
# HotpotQA on a low-RAM machine (corpus subsampled to 2000 docs;
# judged docs are ALWAYS preserved so metrics stay valid)
python run_enhanced_benchmark.py --dataset hotpotqa --max-queries 50 --max-docs 2000
# Tune ColBERT encoding batch size for your RAM
python run_enhanced_benchmark.py --dataset scifact --encode-batch-size 8
# List every available pipeline
python run_enhanced_benchmark.py --help # see the --pipelines choices
# Generate a Markdown report from the latest run
make report

The two old divergent runners (run_benchmark.py, run_full_benchmark.py) were removed in v0.3.0 — one of them mixed single-vector tree embeddings with MaxSim scoring and could only crash. Everything routes through run_enhanced_benchmark.py now (one runner, one shared SBERT/ColBERT/BART instance across all pipelines — see "Memory" notes in the runner docstring).

# Generate triples from BEIR and train (untrained projection is the #1
# reason late interaction underperforms dense — this fixes it)
make train-colbert
# or: python -m src.training.train_colbert --beir-dataset scifact --epochs 3
# Hard-negative mining via BM25 (stronger training signal than random negatives)
python -m src.training.train_colbert --beir-dataset scifact --epochs 5 --hard-negatives
# From pre-generated triples
python -m src.training.train_colbert --triples data/triples.jsonl --epochs 5
# Then benchmark WITH the trained encoder:
make benchmark-trained
# or:
python run_enhanced_benchmark.py --dataset scifact --max-queries 100 \
 --colbert-checkpoint checkpoints/final_model.pt

# Auto-generate report from latest benchmark run
make report
# or: python -m src.eval.report experiments/runs/<run_dir>
# Specific run
python -m src.eval.report experiments/runs/enhanced_scifact_1234567890

# Run with specific config
python run_enhanced_benchmark.py --config configs/scifact.yaml
# Override config values
python run_enhanced_benchmark.py --config configs/full_ablation.yaml --max-queries 50

Centralized in src/config.py. Six nested dataclasses:

from src.config import ExperimentConfig
# Defaults
config = ExperimentConfig.defaults(dataset="scifact")
# With overrides (dot notation)
config = ExperimentConfig.defaults(
 dataset="hotpotqa",
 max_queries=50,
 **{"encoder.projection_dim": 256, "chunking.chunk_size": 100}
)
# From YAML
config = ExperimentConfig.from_yaml("configs/full_ablation.yaml")
# From JSON
config = ExperimentConfig.from_json("experiment_config.json")
# Validate
config.validate() # Checks pipeline names, top_k >= 1, etc.

Pre-built configs: SCIFACT_DEFAULTS, HOTPOTQA_DEFAULTS, FULL_ABLATION.

Three-job GitHub Actions pipeline in .github/workflows/tests.yml:

1. test-core: Python 3.10/3.11/3.12 matrix, installs only numpy/scipy/sklearn/rank-bm25/umap/tqdm, runs tests/run_core_tests.py (no torch)
2. test-full: Python 3.12, installs full requirements, runs pytest tests/ -v
3. benchmark: Python 3.12, runs fast benchmark on SciFact (20 queries, skip heavy), uploads results as artifact

Used to combine rankings from multiple retrieval systems:

RRF_score(d) = Σ 1/(k + rank_i(d))

Where k is typically 60. Each document's score is the sum of reciprocal ranks across all input rankings. Simple, effective, no training needed.

For choosing how many clusters in GMM:

BIC = -2 * log(likelihood) + p * log(n)

Where p is number of parameters and n is number of data points. Penalizes overfitting — BIC prefers simpler models unless the data strongly supports more clusters.

In src/baselines/graph_retrieval.py:

1. Initialize each node as its own community
2. Each node adopts the most common community label among its neighbors (weighted by edge strength)
3. Repeat until convergence (or 10 iterations)
4. Groups of nodes with the same label form communities

Fallback when BART can't load:

1. Split text into sentences
2. Score each sentence: 0.5 * tf_score + 0.3 * position_score + 0.2 * length_score
3. tf_score = average IDF-weighted word frequency across all sentences
4. position_score = 1/(1 + i*0.2) (decay for later sentences)
5. length_score = min(word_count/15, 1.0)
6. Pick top-k sentences, return in original order

See RESEARCH_NOTES.md for detailed technical analysis of 11 research topics with paper links, implementability assessments, and concrete improvement ideas.

MIT