Research map / Large language models / Retrieval and knowledge
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SePer / ΔSePer, a sampling-based semantic perplexity metric that measures retrieval utility in RAG as the reduction in the LLM's belief in ground-truth answers after retrieval.
Problem
Existing RAG evaluation either judges retrieval and generation jointly or uses retriever-only metrics like NDCG, failing to capture retrieval's distinct contribution to generation, especially partial/intermediate information in multi-step RAG; LLM judges are expensive.
Approach
Define retrieval utility as the shift in the LLM's belief (probability) over ground-truth answers with vs. without retrieved context, estimated by Monte-Carlo sampling of responses, semantic clustering via NLI entailment, and exponentiated cross-entropy (semantic perplexity), extending semantic entropy.
Key results
- On EVOUNA answer scoring, NLI-based entailment scoring (SePer) reaches F1/Acc of about 96.6/95.3 on NQ with DPR-FiD, close to GPT-4 and above lexical match and BERTScore.
- ΔSePer_S achieves Pearson correlation with ground-truth retrieval utility of 0.769 on NQ, 0.905 on SQuAD and 0.584 on 2WikiMHQA, outperforming ROUGE, BERTScore, semantic entropy and Prometheus.
- Across 7 entailment models, pairwise Pearson correlations of SePer are above 0.7, with most simple-QA scores above 0.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use soft probability-mass assignment instead of hard semantic-equivalence matching (ablation) | ΔSePer · NQ, MS MARCO, SQuAD, 2WikiMultihopQA, HotpotQA, IIRC, MuSiQue | Pearson correlation with ground-truth retrieval utility: 0.752 -> 0.769; 0.512 -> 0.533; 0.904 -> 0.905; 0.559 -> 0.584; 0.634 -> 0.660; 0.446 -> 0.461; 0.543 -> 0.559 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · NQ | Pearson correlation with ground-truth retrieval utility: 0.691 -> 0.769 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · MS MARCO | Pearson correlation with ground-truth retrieval utility: 0.443 -> 0.533 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · SQuAD | Pearson correlation with ground-truth retrieval utility: 0.808 -> 0.905 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · 2WikiMultihopQA | Pearson correlation with ground-truth retrieval utility: 0.502 -> 0.584 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · HotpotQA | Pearson correlation with ground-truth retrieval utility: 0.508 -> 0.660 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · IIRC | Pearson correlation with ground-truth retrieval utility: 0.399 -> 0.461 | Helped |
| Use ΔSePer_S rather than the strongest reported baseline metric | Llama-2-7B-Chat retrieval utility evaluation · MuSiQue | Pearson correlation with ground-truth retrieval utility: 0.489 -> 0.559 | Helped |
| Increase the number of sampled responses (ablation) | ΔSePer · NQ, HotpotQA, MS MARCO, SQuAD | correlation with ground truth | Helped |
| Use 10 samples rather than 20 (ablation) | ΔSePer · NQ, HotpotQA, MS MARCO, SQuAD | correlation with ground truth: less than a 1% performance drop | Hurt |
| Increase retrieved items from 5 to 10 (ablation) | RAG · AmbigQA | answer quality | Hurt |
| Apply prompt compression (ablation) | RAG · Tested datasets | SePer: about 40% | Hurt |
Surprising findings
- Retrieved items can hurt performance on some datasets (e.g., MS MARCO, MMLU, FEVER with many docs), and reranking slightly hurt NQ and AmbigQA.
- Medium-size generators benefit most from retrieval in the evaluated settings.
Limitations
- Relies on NLI entailment model for semantic clustering, whose accuracy bounds the metric.
- Multi-step utility assumes uniform distribution of overall utility across reasoning steps, which is approximate.
- Correlation with ground-truth utility for reasoning QA is moderate (about 0.45 to 0.58).
Experimental setup
Datasets: Natural Questions TriviaQA MS MARCO SQuAD 2WikiMultihopQA HotpotQA IIRC MuSiQue EVOUNA AmbigQA MMLU FEVER
Benchmarks: EVOUNA Natural Questions MS MARCO SQuAD HotpotQA 2WikiMultihopQA IIRC MuSiQue
Metrics: F1 Accuracy Pearson correlation p-value standard deviation coefficient of variation
Baselines: Exact Match ROUGE BLEU BERTScore Perplexity Semantic Entropy Auto-J Prometheus
Builds on: semantic entropy BERTScore Selective-Context LongLLMLingua
Closest papers by meaning
- RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.90)
- A Theory for Token-Level Harmonization in Retrieval-Augmented Generation (ICLR 2025 · similarity 0.90)
- Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.89)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.89)
- RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation (NeurIPS 2024 · similarity 0.88)
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.