Research map / Label-efficient and robust learning / Recognition and long-tail
Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
Andrew Klearman, Radu Revutchi, Rohin Garg, Rishav Chakravarti, Samuel Marc Denton, Yuan Xue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use coverage-aware stratified query generation instead of original BEIR queries | NFCorpus evaluation set · NFCorpus | Minimum Semantic Coverage: 50.9% -> 90.3% | Improved |
| use coverage-aware stratified query generation | NFCorpus evaluation set · NFCorpus | Zero-query clusters: 182 -> 36 | Improved |
| use coverage-aware stratified query generation | FiQA evaluation set · FiQA | Zero-query clusters: 232 -> 0 | Improved |
| use coverage-aware stratified query generation | SciDocs evaluation set · SciDocs | Minimum Semantic Coverage: 89.2% -> 99.7% | Improved |
| stratify queries by semantic cluster (within-stratum vs overall variance) | Dense retrieval nDCG@10 · NFCorpus | standard deviation: .31 -> .29 | Improved |
| add test queries for under-covered clusters | NFCorpus retrievers · NFCorpus zero-query cluster Statistical methods | nDCG@10: 0.365 / 0.336 / 0.483 (dense / BM25 / hybrid) | Improved |
| use hybrid retrieval instead of dense or BM25 alone | Statistical methods cluster · NFCorpus | nDCG@10: 0.365 -> 0.483 | Improved |
| use BM25 instead of dense retrieval | Cancer cell lines cluster · NFCorpus | nDCG@10: 0.795 -> 0.854 | Improved |
| switch model comparison from overall mean to macro-average across clusters | text-embedding-3-small vs embed-english-v3.0 · NFCorpus, 1,000 bootstrap samples | win rate of text-embedding-3-small: 32.2% -> 63.3% | Mixed |
| lower relevance dispersion (Delta) at fixed alignment | Query regime · NFCorpus | mean nDCG@10: .25 -> .45 (Lo J) | Improved |
| raise query-document semantic alignment (J) at fixed dispersion | Query regime · NFCorpus | mean Recall@100: .22 -> .55 (Hi Delta) | Improved |
Closest papers by meaning
- Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals (NeurIPS 2025 · similarity 0.87)
- SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction (ICLR 2025 · similarity 0.87)
- FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents (NeurIPS 2025 · similarity 0.87)
- Improving Consistency in Retrieval Augmented Systems With Group Similarity Rewards (NeurIPS 2026 · similarity 0.87)
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.87)
- SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
