Research map / Large language models / Retrieval and knowledge
Do Retrieval Augmented Language Models Know When They Don’t Know?
Youchao Zhou, Heyan Huang, Yicheng Liu, Rui Dai, Xinglin Wang, Xingchen Zhang, Shumin Shi, Yang Deng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one positive chunk to an all-negative retrieved context | Vanilla RALM · RGBen calibration | Brier score (Agree): 0.192 -> 0.026 | Improved |
| provide all-negative context instead of no context | Vanilla Qwen-2.5-7B · RALM | refusal rate / overall accuracy: OAcc 0.427 -> 0.290 | Hurt |
| apply R-tuning refusal training | Qwen-2.5-7B · 0p10n context | over-refusal rate: 0.355 -> 0.678 | Hurt |
| apply ICFT(n) in-context fine-tuning with negative context | Qwen-2.5-7B · 0p10n context | over-refusal rate: 0.355 -> 0.270 | Improved |
| apply ICFT(n) | Qwen-2.5-7B · 0p10n context | overall accuracy: 0.290 -> 0.620 | Improved |
| apply ICFT(p) (positive context only) | Qwen-2.5-7B · 1p9n context | overall Brier score: 0.079 -> 0.068 | Improved |
| apply refusal tuning (R-tuning, ICFT) with positive context available | Qwen-2.5-7B · 1p9n context | overall accuracy: 0.863 -> 0.830 (R-tuning), 0.787 (ICFT n) | Hurt |
| apply R-tuning | Qwen-2.5-7B · 0p10n context | denoising rate: 0.191 -> 0.138 | Hurt |
| apply ICFT variants | Qwen-2.5-7B · 0p10n context | denoising rate: 0.191 -> 0.632 to 0.691 | Improved |
| add two-stage knowledge-state aware refusal over threshold post refusal | Vanilla Qwen-2.5-7B · 0p10n context | overall accuracy / over-refusal: OAcc 0.437 -> 0.523; OR 0.770 -> 0.282 | Improved |
| add two-stage refusal | ICFT(n) Qwen-2.5-7B · 0p10n context | overall accuracy: 0.673 -> 0.729 | Improved |
| increase positive chunks beyond one (5p5n) | RALM · RGBen / RGBzh calibration | Brier score (Agree): RGBen 0.026 -> 0.010; RGBzh 0.078 -> 0.150 | Mixed |
Closest papers by meaning
- Utilize the Flow Before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuning (AAAI 2025 · similarity 0.89)
- To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty (NeurIPS 2024 · similarity 0.87)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.87)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.87)
- OR-Bench: An Over-Refusal Benchmark for Large Language Models (ICML 2025 · similarity 0.87)
- Improving Retrieval Augmented Language Model with Self-Reasoning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
