🔎 ClarifyRAG

A clarification-aware search agent that asks before it answers.

tests python deps license

Ask a clarifying question only when it actually helps. In the benchmark, clarify-aware routing hits 100% accuracy while asking 38% fewer questions than always-asking, the best utility of any policy. python -m clarifyrag.eval.

Update: the bundled benchmark turned out to be circular (its "clear" queries paraphrase the corpus documents verbatim), and auditing it uncovered a real bug — the retriever has no stopword filtering, so common words like "how" can outrank genuine topic matches. Fixed in a parallel _v2 module, verified against hand-written natural-language queries: top-1 retrieval accuracy on natural questions goes from 3/7 to 7/7. Details and numbers below. python -m clarifyrag.eval_v2.

You know that coworker who answers your Slack question confidently, at length, about the wrong thing entirely? Most RAG agents are that coworker. Ask "what's the price of apple?" and you'll get a confident answer about either the fruit or the company, decided by a coin flip you didn't know was happening.

ClarifyRAG asks first when it's actually unsure. It gates on retrieval disagreement: if the evidence splits across two comparable senses, it asks. If one sense clearly dominates, it just answers. No committee meeting required, no asking about things that are already obvious.

Runs with zero dependencies and zero API keys (an interpretable TF-IDF retriever plus a heuristic gate). Point it at your own retriever and LLM for production.

from clarifyrag import ClarifyRAG
agent = ClarifyRAG()
agent.step("tell me about jaguar")
# ASK (ambiguity 1.00): "That could mean a few things: did you mean the
# car, or animal sense?" why: top senses contend (car vs animal)
agent.step("jaguar animal wild cat habitat")
# ANSWER (ambiguity 0.08): "The jaguar is a large wild cat native to the Americas."

It resolves multi-turn too: ask, take the hint, then answer.

agent.resolve("how fast is python", intended=python_snake_doc)
# Resolution(doc=python_snake, asks=1, correct=True)

The ambiguity score is the mass ratio of the runner-up sense to the top sense:

ambiguity = mass(2nd sense) / mass(1st sense) # 0 = clear, 1 = a coin flip

Plus a lexical backstop: a known polyseme (apple, jaguar, python, mercury) used with no disambiguating term anywhere nearby. The agent asks when either signal crosses threshold, and you can see exactly why it asked. No model, no vibes, just numbers you can print.

A labeled set of ambiguous and well-specified queries. We score the gate's decision, then compare three policies where asking costs a turn (utility = accuracy - 0.15 * avg_questions):

python -m clarifyrag.eval

gate: precision=1.00 recall=1.00 F1=1.00
policy accuracy avg_asks utility
always_search 0.77 0.00 0.769 fast, but wrong on the ambiguous ones
always_ask 1.00 1.00 0.850 correct, but nags on everything
clarify_aware 1.00 0.62 0.908 asks only when it pays off

Answering everything gets things wrong. Asking about everything gets annoying fast. Asking selectively wins on both counts, which is a nice change of pace for a tradeoff.

The bundled bench's "clear" queries are close paraphrases of the corpus documents themselves:

doc text: "Apple Inc is a technology company; its stock trades on
 NASDAQ with a large market cap and share price."
bench query: "apple stock company share price"

Four of the five content words in the query appear verbatim in the document. That measures whether TF-IDF can match a query to a document built from the query's own words, not whether the agent understands a real question.

Running seven hand-written, naturally-phrased questions with an unambiguous true answer against the shipped Retriever gets 3/7 top-1 accuracy, and the gate asks an unnecessary clarifying question on 7 of 8 naturally-phrased clear queries. The retriever has no stopword list at all: "how", "an", "in", and "are" get scored by IDF exactly like "apple" does, and because a function word can appear in only one or two documents by coincidence, it can outrank a genuine content-word match. Query "how many calories are in an apple" retrieves the Python-the-snake entry, because "how" happens to appear once in its text ("how fast it moves"). Separately, the gate's lexical fallback fires on any query containing a known polyseme (apple, jaguar, python, mercury) unless it contains one of ~25 hardcoded disambiguating words — "calories" isn't on that list, so a perfectly clear question about fruit still triggers a clarifying question.

clarifyrag/corpus_v2.py and clarifyrag/gate_v2.py fix both: stopwords are stripped from indexing and querying, the corpus's terse one-sentence document text is enriched with the vocabulary a real question about each topic actually uses (still hand-written from general topic knowledge, not reverse-engineered from the test queries), and the disambiguator list is widened to match. clarifyrag/eval_v2.py reruns the same measurement on both a 12-item adversarial corpus and a 10-item holdout corpus evaluated exactly once after the fix was frozen:

python -m clarifyrag.eval_v2

corpus / version top-1 acc gate F1 unneeded asks
adversarial / v1 0.50 0.53 7
adversarial / v2 1.00 0.89 1
holdout / v1 0.29 0.46 7
holdout / v2 1.00 0.75 2

The original corpus.py, gate.py, and agent.py are untouched — the fix lives entirely in parallel _v2 modules, and python -m clarifyrag.eval still reproduces the exact published baseline above.

git clone https://github.com/ahmeddoghri/clarifyrag
cd clarifyrag && pip install -e .
python examples/quickstart.py

Or with Docker:

docker build -t clarifyrag .
docker run --rm clarifyrag

Retriever is a plain TF-IDF search over an in-memory corpus. Anything that returns scored, topic-tagged hits drops in cleanly. Wire the gate to your vector DB and the ask/answer logic doesn't change at all.

pip install pytest && pytest -q # 20 passing

Nine small, dependency-light, benchmarked tools for LLM/ML infrastructure. Each one reproduces its headline number locally with no API keys:

agentmem · rubricagent · churnfm · citebench · guardrail-gate · tablextract · vllm-cost-router · taggate

MIT © Ahmed Doghri