Research map / Large language models / Reasoning and chain of thought
Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits how hallucinations emerge and propagate through long chain-of-thought reasoning in DeepSeek-R1 using a controlled RFC-based dataset, and tests seven hallucination detectors.
Problem
Hallucinations in reasoning LLMs can emerge and be amplified across long CoT traces through reflection, yet existing mitigation and detection methods give little insight into how they evolve, especially in black-box settings.
Approach
Builds a controlled RFC-based knowledge domain with Type I (seen but unlearned) and Type II (unseen or incorrect) hallucination subsets and controls, models metacognitive confidence updates during reflection with a prompt-aligned bias assumption, and audits CoT trajectories behaviorally rather than via model internals.
Key results
- Overconfidence leads to hallucination passage rates of 62.54% (Type I) and 56.08% (Type II) in long CoT; the model resists erroneous guidance in only 10.66% of cases.
- Only 22.5% of intervention edits successfully reversed hallucinated outcomes, despite 71.83% of edits being accepted.
- Semantic Entropy achieves the best detection (78.95% accuracy, 85.23 AUROC), but takes days of computation; internal signal probing methods score AUROC below 55%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic consistency checking with Semantic Entropy rather than the other evaluated detectors | Hallucination detection on the controlled RFC audit · Controlled RFC hallucination audit | AUROC: 85.23±0.55 | Helped |
| Use internal signal probing methods | Hallucination detection in reflective CoT · Controlled RFC hallucination audit | AUROC: below 53% | Hurt |
| Add external incorrect facts to the prompt (Type II rather than Type II Control) | DeepSeek-R1 · Type II vs. Type II Control | hallucination acceptance rate: 11.0% -> 56.1% | Mixed |
| Use a Type I seen-but-unlearned setting rather than Type I Control | DeepSeek-R1 · Type I vs. Type I Control | hallucination acceptance rate: 92.6% -> 62.5% | No clear effect |
| Introduce upstream CoT edits rather than later edits | Reasoning trajectories with hallucinations · CoT editing experiments | downstream influence | Helped |
| Edit hallucinated CoT trajectories | Reasoning trajectories with hallucinations · CoT editing experiments | hallucination reversal rate: 22.5% | Helped |
| Increase the number of hallucinations observed across five runs | DeepSeek-R1 Type I reasoning · Type I samples | average CoT length (claims): 26.10 -> 53.31 | Mixed |
| Reason in hallucination cases rather than the correct-answer control | DeepSeek-R1 · Type I vs. Control | average explicit reflections per CoT: 4.40 -> 9.33 | Mixed |
| Reason in hallucination cases rather than the correct-answer control | DeepSeek-R1 · Type I vs. Control | average hedging words per CoT: 16.92 -> 37.14 | Mixed |
| Use the Type II setting rather than the correct-answer control | DeepSeek-R1 · Type II vs. Control | average hallucinated claims per CoT: 0.25 -> 7.01 | Hurt |
Surprising findings
- Hallucinated CoTs can show lower semantic entropy than correct ones, reflecting repeated attempts to correct the same false claim rather than grounded knowledge.
- Even when the model correctly identifies errors in external knowledge, it still fabricates internal supporting knowledge to align with the prompt.
Limitations
- Findings rely on qualitative auditing rather than quantitative confidence estimation, which may introduce bias.
- No hallucination mitigation strategy is explored in black-box settings.
- Main experiments use a single model (DeepSeek-R1); other models were evaluated only on filtered subsets.
Experimental setup
Datasets: RFC documents (314 for Type I, 50 for Type II and control)
Benchmarks: Controlled Hallucination Audit Dataset
Metrics: Accuracy Recall F1 Score AUROC acceptance rate hallucination rate
Baselines: Logit Entropy Attention Strength Spectral Entropy HDM2 CCP SelfCheckGPT Semantic Entropy
Builds on: chain-of-thought prompting SelfCheckGPT Semantic Entropy
Closest papers by meaning
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models (NeurIPS 2025 · similarity 0.93)
- MIRAGE: Assessing Hallucination in Multimodal Reasoning Chains of MLLM (NeurIPS 2025 · similarity 0.93)
- Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models (AAAI 2026 · similarity 0.92)
- Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models (CVPR 2026 · similarity 0.92)
- Attributive Reasoning for Hallucination Diagnosis of Large Language Models (AAAI 2025 · similarity 0.91)
- Mitigating Hallucinations in Large Language Models via Causal Reasoning (AAAI 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.