Research map / Large language models / Retrieval and knowledge
RaDIO: Real-Time Hallucination Detection with Contextual Index Optimized Query Formulation for Dynamic Retrieval Augmented Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaDIO, a dynamic RAG framework that uses an MLP hallucination detector on LLM hidden states to decide when to retrieve and a multi-head-attention/TF-IDF/position/embedding-similarity token scoring to decide what to query.
Problem
Dynamic RAG methods use static rules to decide when to retrieve and crude last-sentence or last-tokens heuristics to form queries, causing unnecessary or poor retrievals during long-form generation.
Approach
Trains an unsupervised MLP hallucination detector on Wikipedia-truncation continuations (labeled by entity mismatch) that monitors token-level final-layer states in real time, and formulates queries by ranking tokens with a combined score of attention, TF-IDF, relative position, and embedding cosine similarity to the query.
Key results
- On 2WikiMultihopQA with SBERT retrieval, RaDIO reaches EM 0.254 and F1 0.317 versus DRAGIN's 0.231 and 0.294.
- On HotpotQA with BM25, RaDIO improves EM from 0.219 to 0.246 and F1 from 0.314 to 0.351 over DRAGIN.
- On StrategyQA, wo-RAG (precision 0.659) outperforms RaDIO (0.654), a reported exception to RaDIO's gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RaDIO instead of DRAGIN | LLaMA2-7B-CHAT · 2WikiMultihopQA | EM: 0.231 -> 0.254 | Helped |
| Use RaDIO instead of DRAGIN | LLaMA2-7B-CHAT · 2WikiMultihopQA | F1: 0.294 -> 0.317 | Helped |
| Use RaDIO instead of DRAGIN | LLaMA2-7B-CHAT · HotpotQA | EM: 0.219 -> 0.246 | Helped |
| Use RaDIO instead of DRAGIN | LLaMA2-7B-CHAT · HotpotQA | F1: 0.314 -> 0.351 | Helped |
| Use RaDIO instead of wo-RAG | LLaMA2-7B-CHAT · StrategyQA | Pre.: 0.659 -> 0.654 | Hurt |
| Use RaDIO instead of the strongest listed baseline, SR-RAG | LLaMA2-7B-CHAT · IIRC | EM: 0.187 -> 0.196 | Helped |
| Use RaDIO instead of the strongest listed baseline, SR-RAG | LLaMA2-7B-CHAT · IIRC | F1: 0.226 -> 0.239 | Helped |
Surprising findings
- Static-rule RAG methods (SR-RAG, FL-RAG) underperform no-retrieval wo-RAG on HotpotQA, suggesting poorly timed retrieval hurts quality.
- On StrategyQA, the no-retrieval baseline beats all RAG methods including RaDIO.
Limitations
- Evaluated only with LLaMA2-7B-Chat, so generalization to other LLMs is untested.
- Hallucination detector training set size and MLP architecture details are not specified.
- Reported gains are modest in several settings and the paper lacks significance testing.
Experimental setup
Datasets: Wikipedia (hallucination training data) 2WikiMultihopQA HotpotQA StrategyQA IIRC
Benchmarks: 2WikiMultihopQA HotpotQA StrategyQA IIRC
Metrics: EM F1 Precision Recall Retrieve Count Generate Count Hallucinated Count Token Count Sentence Count
Baselines: DRAGIN wo-RAG SR-RAG FL-RAG FS-RAG FLARE
Builds on: DRAGIN FLARE IRCoT IC-RALM
Closest papers by meaning
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability (ICLR 2025 · similarity 0.89)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.88)
- Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling (AAAI 2025 · similarity 0.88)
- Teaching Language Models to Hallucinate Less with Synthetic Tasks (ICLR 2024 · similarity 0.87)
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation (CVPR 2024 · similarity 0.87)
- HaDeMiF: Hallucination Detection and Mitigation in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.