Research map / Image generation / Fast diffusion sampling
Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Probes why diffusion models hallucinate text: a "local generation bias" measured by a Local Dependency Ratio (LDR), with a theoretical two-layer ReLU analysis on hypercube parity distributions.
Problem
Diffusion models correctly render individual symbols but assemble them nonsensically (text hallucination); the mechanism behind this failure is unclear.
Approach
Identifies a local generation bias in score networks: when symbol dimensions are nearly pairwise independent, denoisers rely on local regions, generating each symbol's marginal independently. Proposes the Local Dependency Ratio (LDR) probe to measure this and gives a gradient-flow analysis showing small-initialized two-layer ReLU networks converge toward LDR≈1 invariant sets.
Key results
- On Quarter-MNIST, UNet reaches only 20.6% rule-satisfying samples while DiT reaches about 90%; neither generates valid tuples outside the training set (<0.5%).
- UNet's LDR for the top-left region stays above 0.75 throughout training, while DiT's LDR is high early and drops around 30k-50k iterations, coinciding with accuracy gains.
- For parity parenthesis with L=16, the model generates correct samples only by chance, and about half of samples are hallucinations early in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiT instead of UNet | Quarter-MNIST diffusion generation · Quarter-MNIST | valid addition accuracy: 20.6% compared with ~ 90% | Helped |
| Generate symbols with local dependencies rather than modeling global symbol relations | Diffusion models on parity-constrained sequences · Parity Parenthesis, L=8 | hallucinated sample fraction: 25% | Hurt |
| Train on Quarter-MNIST addition constraints without prior knowledge | UNet and DiT · Quarter-MNIST test combinations | valid tuple generation fraction beyond training data: less than 0.5% | Hurt |
| Use an attention-capable DiT, whose LDR decreases at 30k to 50k iterations | Quarter-MNIST diffusion generation · Quarter-MNIST | valid addition accuracy: 30k to 50k | Helped |
| Train the UNet on Quarter-MNIST | UNet · Quarter-MNIST | LDR: more than $0.75$ | Hurt |
| Train diffusion models on real-world English-word and Chinese-character distributions | Diffusion models on real-world text · English words and Chinese characters | hallucination: 1,000 common English words; 1,000 common Chinese characters | Mixed |
Surprising findings
- Local generation bias persists even in transformer (DiT) and attention-equipped UNet architectures with global receptive fields.
- Generated samples never extrapolate beyond the training set; diffusion models only oscillate between hallucination and overfitting for the synthetic grammar.
Limitations
- Theoretical analysis is restricted to a two-layer ReLU network on the hypercube with |S|=2 and specific grammar assumptions (pairwise-independent marginals).
- Real-world LDR analysis of FLUX.1 and SD3.5 uses a zeroth-order approximation rather than the exact Jacobian.
Experimental setup
Datasets: MNIST synthetic parity parenthesis images rendered English words rendered Chinese characters
Benchmarks: Quarter-MNIST synthetic parity parenthesis
Metrics: fraction of samples satisfying grammar rule (OCR-based) Local Dependency Ratio (LDR)
Baselines: UNet DiT MLP
Builds on: denoising diffusion probabilistic models score-based generative models DiT saliency maps
Closest papers by meaning
- Understanding Hallucinations in Diffusion Models through Mode Interpolation (NeurIPS 2024 · similarity 0.90)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.88)
- Sample Transform Cost-Based Training-Free Hallucination Detector for Large Language Models (NeurIPS 2026 · similarity 0.88)
- Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations (NeurIPS 2025 · similarity 0.87)
- Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.87)
- When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.