Research map / Architectures and efficiency / RNNs and normalization
From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses sparse autoencoders on residual streams of CLIP ViT-B/32, Pythia-160m, and Gemma 2B to show transformers impose coherent concepts on noise/shuffled inputs, and that input-side concept activations predict output hallucination via PLS regression.
Problem
How and when hallucinations arise in pre-trained transformers, and whether internal concept representations of the input reflect input structure or model-imposed priors under uncertain or noisy inputs.
Approach
Train SAEs on residual-stream activations of pure Gaussian noise inputs and on perturbed (patch- or n-gram-shuffled) inputs, measure concept purity and steerability, and show that the number of active concepts grows with input unstructuredness and that input-prompt concept activations linearly predict summary hallucination scores.
Key results
- Layer 11 residual SAE concepts predict Gemma 2B-IT summary hallucination scores with out-of-sample R^2 of 0.271 ± 0.010 (10-fold CV); binary median split accuracy 73.0% ± 5.3%.
- Suppressing the top 10 VIP concepts in Gemma 2B-IT layer 11 lowers mean hallucination score on the top-quartile most hallucinated examples (n=252) from 0.91 to 0.72.
- Image SAE L0 peaks at layer 6 with 38 extra concepts above baseline for 28x28 patch shuffling; text SAE L0 peaks at layer 9 with 81 extra concepts for single-word shuffling.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Suppress the 10 layer 11 SAE concepts with the highest VIP scores | Gemma 2B-IT summary generation · Top quartile most hallucinated Vectara examples | mean hallucination score: from 0.91 to 0.72; mean score drop of 0.19 | Helped |
| Use layer 11 SAE concept activations from the source prompt to predict summary hallucination | Gemma 2B-IT · Vectara summaries (1,006 articles), 10-fold cross-validation | R²: 0.271 ± 0.010 | Helped |
| Use layer 11 SAE concept activations from the source prompt for above/below-median classification | Gemma 2B-IT · Vectara summaries (1,006 articles), 10-fold cross-validation | accuracy: 73.0% ± 5.3% | Helped |
| Use layer 18 SAE source concepts to predict continuous summary hallucination | Gemma 2B-IT · Vectara summaries | R² | No clear effect |
| Shuffle images into 28x28 pixel patches instead of using natural images | CLIP-ViT-B/32 layer 6 SAE activations · Image inputs | SAE L0: 966.36 ± 1.02 -> 1004.33 ± 6.45 | Mixed |
| Shuffle text into single-word sections instead of using natural text | Pythia-160m-deduped layer 9 SAE activations · Text inputs | SAE L0: 67.55 ± 0.24 -> 149.03 ± 1.53 | Helped |
| Train noise SAEs with different random seeds | CLIP-ViT-B/32 noise SAE concepts · Layers 1–3 | Jaccard concept overlap: 35% | Mixed |
| Train noise SAEs with different random seeds | CLIP-ViT-B/32 noise SAE concepts · Layers 5–6 | Jaccard concept overlap: 14% | Mixed |
| Train noise SAEs with different random seeds | CLIP-ViT-B/32 noise SAE concepts · Deeper vision-transformer layers | Jaccard concept overlap: 25% | Mixed |
| Use layer 6 L0 rather than PLS concept patterns to predict image misclassification | CLIP-ViT-B/32 · Natural ImageNet images | AUC: 64.8 ± 1.1 vs. 64.5 ± 1.5 | No clear effect |
| Use layer 6 L0 rather than PLS concept patterns to predict image misclassification | CLIP-ViT-B/32 · 56x56 patch-shuffled images | AUC: 61.3 ± 1.6 vs. 61.5 ± 1.3 | No clear effect |
Surprising findings
- SAEs trained only on pure Gaussian noise activations yield interpretable, steerable concepts (e.g., dogs, sharks, elephants) when probed with ImageNet images.
- Concept overlap between noise SAEs trained with different seeds dips in middle layers, inversely to the L0 increase for unstructured inputs.
Limitations
- Experiments mainly on small models (CLIP ViT-B/32, Pythia-160m, Gemma 2B); not verified on 10-100B+ frontier models.
- Concept evaluation limited by ImageNet-1k validation set, so interpretability and steerability counts are understated.
- Hallucination proxy is faithfulness of summaries measured by HHEM-2.1, not factual correctness; SAEs are not perfect oracles.
Experimental setup
Datasets: ImageNet-1k FineWeb-Edu Vectara hallucination leaderboard TruthfulQA TriviaQA CoQA
Benchmarks: ImageNet-1k validation Vectara hallucination leaderboard TruthfulQA TriviaQA CoQA
Metrics: semantic purity steerability L0 coefficient of determination AUC accuracy
Baselines: TruthfulQA hallucination detection baselines TriviaQA hallucination detection baselines CoQA hallucination detection baselines
Builds on: sparse autoencoders linear representation hypothesis Gemma Scope
Closest papers by meaning
- InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration (AAAI 2026 · similarity 0.88)
- Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models (NeurIPS 2025 · similarity 0.88)
- In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation (ICML 2024 · similarity 0.87)
- GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs (ICLR 2026 · similarity 0.87)
- On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models (NeurIPS 2025 · similarity 0.87)
- Dual-Pathway Circuits of Object Hallucination in Vision-Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.