ReRag: a Reconfigurable Retrieval-Augmented-Generation Experimentation and Validation framework

🚀 Quick Start

Prerequisites

1. Setup Environment

# Clone repository
git clone <repository-url>
cd ReRag
# Create virtual environment
python -m venv venv
source venv/bin/activate # Linux/Mac
# venv\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt
# Configure API keys
cp .env_example .env
# Edit .env and add your API keys:
# GOOGLE_API_KEY=your_key_here
# OPENAI_API_KEY=your_key_here

2. Start Vector Database

# Start Qdrant
docker-compose up -d
# Verify it's running
curl http://localhost:6333/healthz
#You can see the ingestion results in Qdrant's Web UI visiting the link below:
http://localhost:6333/dashboard#/collections

3. Run Your First Pipeline

#First download the dataset from the scripts folder
# Ingest documents (requires dataset - see Data Ingestion section)
python bin/ingest.py ingest --config pipelines/configs/datasets/stackoverflow_hybrid.yml
# Run agent in interactive mode
python main.py
# Run agent with single query
python main.py --query "What are Python best practices?"
# Run Self-RAG mode (with iterative refinement)
python main.py --mode self-rag --query "Explain how asyncio works"

📚 User Guide

Data Ingestion

# Basic ingestion from config
python bin/ingest.py ingest --config pipelines/configs/datasets/stackoverflow_hybrid.yml
# Test with dry run (no upload)
python bin/ingest.py ingest --config my_config.yml --dry-run --max-docs 100
# Check ingestion status
python bin/ingest.py status
# Cleanup canary collections
python bin/ingest.py cleanup

dataset:
 name: "my_dataset"
 adapter: "stackoverflow" # or full path: "pipelines.adapters.custom.MyAdapter"
 path: "datasets/sosum/data"
embedding:
 strategy: "hybrid" # or "dense" or "sparse"
 dense:
 provider: "google"
 model: "text-embedding-004"
 sparse:
 provider: "sparse"
 model: "Qdrant/bm25"
qdrant:
 collection: "my_collection"
 host: "localhost"
 port: 6333

Retrieval Testing

# Use any retrieval configuration
python bin/retrieval_pipeline.py \
 --config pipelines/configs/retrieval/basic_dense.yml \
 --query "How to handle Python exceptions?" \
 --top-k 5

Agent Workflows

# Standard RAG mode (single-pass)
python main.py --query "Explain Python decorators"
# Self-RAG mode (iterative refinement with verification)
python main.py --mode self-rag --query "How does asyncio work?"
# Interactive chat
python main.py
# or
python main.py --mode self-rag

Benchmarking

# Run experiment with output directory
python -m benchmarks.experiment1 --output-dir results/exp1
# Run 2D grid optimization for hybrid search parameters
python -m benchmarks.optimize_2d_grid_alpha_rrfk \
 --scenario-yaml benchmark_scenarios/your_scenario.yml \
 --dataset-path datasets/sosum/data \
 --n-folds 5 \
 --output-dir results/optimization
# Generate ground truth for evaluation
python -m benchmarks.generate_ground_truth \
 --queries-file queries.json \
 --output-file ground_truth.json

📖 System Architecture

Overview

┌────────────────────────────────────────────────────────────┐
│ RAG System │
├────────────────────────────────────────────────────────────┤
│ │
│ 📊 INGESTION → 🔍 RETRIEVAL → 🤖 AGENT │
│ │
│ Documents Vector Search LangGraph │
│ Chunking Reranking Response Gen │
│ Embedding Filtering Verification │
│ ↓ ↓ ↓ │
│ └───────────→ Qdrant ←───────────┘ │
│ │
│ 📈 BENCHMARKS: Evaluation & Optimization │
└────────────────────────────────────────────────────────────┘

Core Components

🔧 Installation

1. Python Environment

# Clone repository
git clone <repository-url>
cd Thesis
# Create virtual environment (Python 3.11+ required)
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt

2. API Keys

# Create environment file
cp .env_example .env

# Required
GOOGLE_API_KEY=your_key_here
OPENAI_API_KEY=your_key_here
# Optional
VOYAGE_API_KEY=your_key_here

3. Start Vector Database

# Start Qdrant using Docker
docker-compose up -d
# Verify it's running
curl http://localhost:6333/health

📁 Project Structure

Thesis/
├── readme.md # This file
├── main.py # Agent entry point (Standard & Self-RAG modes)
├── config.yml # Main configuration file
├── docker-compose.yml # Qdrant database setup
├── requirements.txt # Python dependencies
│
├── agent/ # LangGraph agent workflows
│ ├── graph_refined.py # Standard RAG workflow
│ ├── graph_self_rag.py # Self-RAG workflow (iterative refinement)
│ ├── schema.py # State definitions
│ └── nodes/ # Agent nodes (retriever, generator, grader)
│
├── pipelines/ # Data ingestion
│ ├── adapters/ # Dataset adapters (StackOverflow, custom)
│ ├── ingest/ # Ingestion pipeline core
│ ├── eval/ # Retrieval evaluator
│ └── configs/ # Dataset configurations
│ └── datasets/ # Per-dataset configs
│
├── components/ # Retrieval pipeline components
│ ├── retrieval_pipeline.py # Pipeline orchestration
│ ├── rerankers.py # CrossEncoder, Semantic, ColBERT, MultiStage
│ ├── filters.py # Tag, duplicate, relevance filters
│ └── post_processors.py # Result enhancement & limiting
│
├── retrievers/ # Core retrieval implementations
│ ├── dense_retriever.py # Dense/sparse/hybrid retrieval
│ └── base.py # Abstract interfaces
│
├── embedding/ # Embedding providers
│ ├── factory.py # Provider factory
│ ├── providers/ # Google, OpenAI, Voyage, HuggingFace
│ └── base_embedder.py # Abstract interfaces
│
├── database/ # Vector database
│ ├── qdrant_controller.py # Qdrant integration
│ └── base.py # Abstract interfaces
│
├── config/ # Configuration system
│ ├── config_loader.py # YAML config loader
│ └── llm_factory.py # LLM provider factory
│
├── benchmarks/ # Evaluation framework
│ ├── experiment1.py # Main experiment runner
│ ├── optimize_2d_grid_alpha_rrfk.py # Grid search optimization
│ ├── llm_as_judge_eval.py # LLM-based evaluation
│ ├── generate_ground_truth.py # Ground truth generation
│ ├── benchmarks_runner.py # Core benchmark runner
│ ├── benchmarks_metrics.py # Metrics (Recall, Precision, MRR, NDCG)
│ ├── report_generator.py # Report generation (used by experiments)
│ └── statistical_analyzer.py # Statistical analysis
│
├── bin/ # CLI tools
│ ├── ingest.py # Ingestion CLI
│ ├── retrieval_pipeline.py # Retrieval testing CLI
│ ├── qdrant_inspector.py # Database inspection
│ └── switch_agent_config.py # Config switcher
│
├── logs/ # Application logs
│ ├── agent.log # Main agent log
│ ├── ingestion.log # Ingestion log
│ └── utils/logger.py # Custom logger
│
└── tests/ # Test suite
 ├── test_self_rag_integration.py # Self-RAG integration tests
 └── [other test files]

⚙️ Configuration

Configuration Files

dataset:
 name: "stackoverflow"
 adapter: "stackoverflow" # or full path
 path: "datasets/sosum/data"
embedding:
 strategy: "hybrid" # dense, sparse, or hybrid
 dense:
 provider: "google"
 model: "text-embedding-004"
 sparse:
 provider: "sparse"
 model: "Qdrant/bm25"
qdrant:
 collection: "my_collection"
 host: "localhost"
 port: 6333

Environment Variables

🔌 Extension Points

Add Custom Dataset Adapter

Add Custom Reranker

from components.rerankers import BaseReranker
class MyReranker(BaseReranker):
 def rerank(self, query: str, results: List[SearchResult]) -> List[SearchResult]:
 # Your reranking logic
 return reranked_results

Add Custom Agent Node

🎯 Key Features

Retrieval Strategies

Reranking

Agent Modes

Benchmarking

📊 Testing

Run Integration Tests

# Self-RAG integration tests
pytest tests/test_self_rag_integration.py -v
# All tests
pytest tests/ -v

Verify Components

🔍 CLI Tools

📈 System Requirements

📚 Documentation

🛠️ Technologies

📧 Contact