Research map / Large language models / Retrieval and knowledge
SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuRe, a zero-shot prompting framework that generates answer candidates, writes candidate-conditioned summaries of retrieved passages, and selects the answer by instance-wise validity and pairwise ranking of those summaries, improving open-domain QA with black-box LLMs.
Problem
Improving open-domain question answering with retrieval-augmented LLMs when only black-box API access is available and fine-tuning is infeasible; naive prompting underuses retrieved passages.
Approach
Generate a few answer candidates, then summarize retrieved passages conditioned on each candidate, and select the final answer by verifying each summary's validity for its candidate and ranking summaries pairwise for informativeness, all via zero-shot prompting.
Key results
- Average EM over four ODQA datasets with ChatGPT and BM25 rises from 26.6 (Base) to 31.2 with SuRe (+4.6 EM, +4.0 F1).
- With LLaMA2-chat-70B on NQ*/WebQ*/2Wiki/HotpotQA, average EM improves from 19.7 to 27.6 with SuRe.
- In human preference on 84 NQ* samples, SuRe summaries were preferred 43.4% vs 26.9% for generic summaries (rest ties).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace appended retrieved passages with SuRe's candidate-conditioned summaries and summary verification | ChatGPT + BM25 · NQ, WebQ, 2Wiki, HotpotQA average | EM: 26.6 -> 31.2 | Helped |
| Replace appended retrieved passages with SuRe's candidate-conditioned summaries and summary verification | ChatGPT + BM25 · NQ, WebQ, 2Wiki, HotpotQA average | F1: 36.1 -> 40.1 | Helped |
| Use SuRe instead of passage-appending prompting | ChatGPT + BM25 · NQ, WebQ, 2Wiki, HotpotQA | EM: 26.6 -> 31.2 | Helped |
| Use SuRe instead of passage-appending prompting | GPT-4 + BM25 · NQ, WebQ, 2Wiki, HotpotQA average | EM: 30.3 -> 33.2 | Helped |
| Use SuRe instead of passage-appending prompting | LLaMA2-chat + BM25 · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 19.7 -> 27.6 | Helped |
| Add candidate-conditioned summaries to passage-appending prompting (ablation) | ChatGPT + BM25 · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 26.8 -> 28.4 | Helped |
| Add pair-wise ranking of summaries (ablation) | Conditional summarizations · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 28.4 -> 28.5 | Helped |
| Add instance-wise validity to pair-wise summary ranking (ablation) | ChatGPT + BM25 · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 28.5 -> 31.2 | Helped |
| Use candidate-conditioned summarization and comparison instead of generic summarization (ablation) | ChatGPT + BM25 · NQ*, WebQ*, 2Wiki, HotpotQA | EM: 27.3 -> 31.2 | Helped |
| Use generated answer candidates as multiple-choice options instead of verifying summaries (ablation) | ChatGPT + BM25 · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 29.8 -> 31.2 | Hurt |
| Increase candidate count from K=2 to K=3 (ablation) | Oracle candidate selection · NQ*, WebQ*, 2Wiki, HotpotQA average | EM: 40.2 -> 41.1 | Helped |
| Use few-shot examples instead of zero-shot prompting (ablation) | ChatGPT + BM25, Base · NQ* | EM: 29.4 -> 31.9 | Helped |
Surprising findings
- Simple Base prompting outperforms several sophisticated baselines such as Selection-inference, chain-of-thought and self-verification in this zero-shot black-box setting.
- Increasing retrieved passages to 50 with Base costs more than SuRe with 10 passages yet achieves lower accuracy on NQ*/WebQ*.
Limitations
- Multiple LLM calls per question make the method costly, with re-reading passages for summaries accounting for most of the cost.
- Gap to oracle over candidates remains large (about 9 EM points), so candidate selection is a bottleneck.
- Long-form QA gains are modest and prompts were designed on short-answer datasets.
Experimental setup
Datasets: Natural Questions WebQuestions 2WikiMultiHopQA HotpotQA 21M English Wikipedia dump ASQA
Benchmarks: Natural Questions WebQuestions 2WikiMultiHopQA HotpotQA ASQA
Metrics: exact match F1 MAUVE LLM-acc ROUGE-L STR-EM
Baselines: No retrieval Base Rerank RePlug Selection-inference Chain-of-thoughts Self-verification
Builds on: chain-of-thought Self-verification Selection-inference RePlug
Closest papers by meaning
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.91)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.90)
- Explore What LLM Does Not Know in Complex Question Answering (AAAI 2025 · similarity 0.89)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.88)
- NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.88)
- Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.