Research map / Large language models / Retrieval and knowledge
How Language Model Hallucinations Can Snowball
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that ChatGPT and GPT-4 often state a wrong answer and then justify it with claims they recognize as false when asked separately, a phenomenon called hallucination snowballing, across three synthetic yes/no QA datasets.
Problem
Language models hallucinate even on simple questions they can recognize as wrong; the paper asks whether hallucinations arise only from knowledge gaps or from over-committing to earlier generated errors.
Approach
Construct three yes/no datasets where the correct answer is fixed and sequential reasoning is hard for one-step transformers, so the model commits to a wrong first token and then generates a justification with incorrect claims; then re-query the same model in a new session to check whether it recognizes those claims as false.
Key results
- ChatGPT detects 67.37% of its snowballed hallucinations and GPT-4 detects 87.03% when re-queried in a separate session.
- GPT-4 has a 83.40% average error rate across the three tasks versus 60.13% for ChatGPT under the direct prompt.
- With 'Let's think step by step', snowballed hallucinations still occur in 94.90% of GPT-4's incorrect cases on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| ask ChatGPT to identify its own incorrect claims in isolation | ChatGPT · All three QA datasets | snowballed hallucination detection rate: 67.37% | Helped |
| ask GPT-4 to identify its own incorrect claims in isolation | GPT-4 · All three QA datasets | snowballed hallucination detection rate: 87.03% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | ChatGPT · Graph Connectivity | error rate: 82.0% -> 27.8% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | ChatGPT · Primality Testing | error rate: 67.8% -> 0.4% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | ChatGPT · Senator Search | error rate: 30.6% -> 0.0% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | GPT-4 · Graph Connectivity | error rate: 88.4% -> 4.2% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | GPT-4 · Primality Testing | error rate: 74.8% -> 7.4% | Helped |
| append “Let's think step-by-step” to the prompt (ablation) | GPT-4 · Senator Search | error rate: 87.0% -> 0.0% | Helped |
| use step-by-step prompting rather than the original prompt (ablation) | GPT-4 · Graph Connectivity and Primality Testing | snowballed hallucination rate: 94.3% -> 95.2%; 92.5% -> 94.6% | No clear effect |
| increase decoding temperature (ablation) | ChatGPT and GPT-4 · Graph Connectivity, Primality Testing, and Senator Search | error rate: ChatGPT average: 60.13% -> 58.53% -> 58.53%; GPT-4 average: 83.40% -> 82.53% -> 81.67% | No clear effect |
| increase decoding temperature (ablation) | ChatGPT and GPT-4 · Graph Connectivity, Primality Testing, and Senator Search | snowballed hallucination rate: ChatGPT average: 67.37% -> 66.77% -> 66.77%; GPT-4 average: 87.03% -> 86.13% -> 84.87% | No clear effect |
Surprising findings
- GPT-4 performs worse than ChatGPT on all three datasets despite being considered the stronger model.
- Zero-shot chain-of-thought prompting reduces errors but still produces snowballed hallucinations in most failing cases.
Limitations
- Only two proprietary models (ChatGPT and GPT-4) were evaluated, with no access to logits or finetuning.
- Recognition heuristics were tuned by manual inspection and may not transfer to other models.
- Limited to English yes/no QA on synthetic tasks; other tasks such as summarization or code are not explored.
Experimental setup
Benchmarks: Graph Connectivity Primality Testing Senator Search
Metrics: error rate snowballed hallucination rate accuracy
Baselines: Let's think step by step zero-shot CoT
Builds on: chain-of-thought prompting exposure bias
Closest papers by meaning
- FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees (ICML 2025 · similarity 0.91)
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.90)
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations (NeurIPS 2025 · similarity 0.89)
- LLM-Check: Investigating Detection of Hallucinations in Large Language Models (NeurIPS 2024 · similarity 0.89)
- Attributive Reasoning for Hallucination Diagnosis of Large Language Models (AAAI 2025 · similarity 0.89)
- In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.