-
Notifications
You must be signed in to change notification settings - Fork 102
feat: surface rank-sensitive retrieval metrics in slice-1 bench (OFFICE-671) - #1130
Conversation
Implement scoreNDCG/scoreRecallAtK/scoreMRR and surface recall@1/@3, nDCG@10, MRR in FormatReport. Adds the retrieval observability layer (OFFICE-671) so reranker deltas are visible in rank-sensitive metrics that recall@20 cannot move. Includes unit tests for the scorers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Important
Review skipped
Auto reviews are disabled on base/target branches other than the default branch.
Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 00116688-b869-477a-9eb4-486cc1f089c7
You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.
Use the checkbox below for a quick retry:
- 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
rag/office-671-retrieval-observability
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
@FranDias
FranDias
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Auto-approved so CI is the gate — review the checks before merging.
@FranDias
FranDias
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Auto-approved so CI is the gate — review the checks before merging.
f0cf0ba to
7325045
Compare
What
Finishes the retrieval observability layer for the slice-1 bench. The rank-sensitive metric fields (
NDCG10,MRR,RecallAt5/10) were wired into the structs and call sites in a prior turn but the scorers were never implemented and the report never printed them — so the runner package did not even build.This PR:
scoreNDCG,scoreRecallAtK,scoreMRRin a newscoring.go.FormatReport.scoring_test.go).Why
bench/slice-1previously computed only recall@20 — a coarse in-set gate that a reranker swap (OFFICE-464) cannot move. The reranker delta lives in rank-sensitive metrics. This layer is the measurement substrate the ranking work depends on.Test plan
go build ./bench/slice-1/...— passesgo test ./bench/slice-1/runner/— all 18 subtests passgofmt— cleanPart of OFFICE-459 (RAG gaps plan). Closes OFFICE-671.
🤖 Generated with Claude Code