Research map / Image generation / Fast diffusion sampling
Interpretable Diffusion via Information Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that diffusion denoisers give exact mutual information, conditional MI and a non-negative pixel-wise decomposition, then uses these to evaluate compositionality (ARO), unsupervised word localization, and prompt-intervention effects with Stable Diffusion 2.1.
Problem
Diffusion models capture complex text-image relationships, but what they learn is opaque; attention maps are a poor proxy for whether a word actually affects the output, and existing information decompositions are intractable in high dimensions.
Approach
Exact expressions for (conditional) mutual information and pointwise information are derived from the MMSE denoiser via the I-MMSE relation, yielding a non-negative per-pixel decomposition i^o_j from the orthogonality principle that quantifies which image variables carry information about which words.
Key results
- On ARO, the information estimator with Stable Diffusion 2.1 reaches 72.0% on VG-A, 69.1% on VG-R, 40.1% on COCO and 49.3% on Flickr30k, outperforming OpenCLIP (64.6, 51.4, 32.8, 40.5).
- For unsupervised object segmentation on COCO-IT at 100 steps, Attention+Information reaches 42.71 mIoU versus 34.90 for attention alone, while pixel-wise CMI alone reaches 33.24.
- Image-level CMI correlates better with the L2 effect of prompt omission than attention does, per the paper's intervention analysis on COCO100-IT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diffusion-based information scoring rather than the strongest reported baseline | Stable Diffusion 2.1 · ARO VG-A | accuracy (%): 67.6 -> 72.0 | Helped |
| Use diffusion-based information scoring rather than the strongest reported baseline | Stable Diffusion 2.1 · ARO VG-R | accuracy (%): 52.3 -> 69.1 | Helped |
| Use diffusion-based information scoring rather than the strongest reported baseline | Stable Diffusion 2.1 · ARO COCO | accuracy (%): 34.4 -> 40.1 | Helped |
| Use diffusion-based information scoring rather than the strongest reported baseline | Stable Diffusion 2.1 · ARO Flickr30k | accuracy (%): 48.6 -> 49.3 | Helped |
| Combine information with attention | Attention-based segmentation on Stable Diffusion · COCO-IT, 50 steps | mIoU (%): 34.52 -> 42.46 | Helped |
| Use conditional mutual information rather than attention for object segmentation | Stable Diffusion · COCO-IT, 100 steps | mIoU (%): 34.90 -> 33.24 | Hurt |
| Use logistic rather than uniform SNR importance sampling (ablation) | Information scoring with Stable Diffusion 2.1 · ARO VG-A | accuracy (%): 71.2 -> 72.0 | Helped |
| Use logistic rather than uniform SNR importance sampling (ablation) | Information scoring with Stable Diffusion 2.1 · ARO VG-R | accuracy (%): 68.5 -> 69.1 | Helped |
| Use logistic rather than uniform SNR importance sampling (ablation) | Information scoring with Stable Diffusion 2.1 · ARO COCO | accuracy (%): 39.3 -> 40.1 | Helped |
| Use logistic rather than uniform SNR importance sampling (ablation) | Information scoring with Stable Diffusion 2.1 · ARO Flickr30k | accuracy (%): 48.7 -> 49.3 | Helped |
Surprising findings
- Pixel-wise CMI is less effective than attention for object segmentation, while MI is more informative than attention for abstract words such as adjectives and adverbs.
- Attention to a word does not imply the word affects the generated image; words with low CMI can be ignored under intervention.
Limitations
- Estimator is not a guaranteed upper or lower bound because neural denoisers may not reach the MMSE optimum.
- Pixel-wise CMI underperforms attention for object segmentation.
- Small prompt changes can cause global image changes, making localized effect correlation poor.
Experimental setup
Datasets: ARO (VG-Relation, VG-Attribution, COCO-Order, Flickr30k-Order) MSCOCO 2017 validation (COCO-IT, COCO100-IT, COCO-WL)
Benchmarks: ARO COCO-IT segmentation (mIoU) COCO100-IT intervention correlation
Metrics: accuracy mIoU Pearson correlation standard error
Baselines: OpenCLIP DiffITM HardNeg-DiffITM DAAM attention MMSE-based scoring
Builds on: I-MMSE relation DAAM ARO benchmark Stable Diffusion
Closest papers by meaning
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.92)
- Feature Information Dynamics in Diffusion (NeurIPS 2026 · similarity 0.90)
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.89)
- CleanDIFT: Diffusion Features without Noise (CVPR 2025 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- How Diffusion Models Memorize (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.