Research map / Image generation / Text-to-image diffusion
Diffusion PID: Interpreting Diffusion via Partial Information Decomposition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffusionPID, applying partial information decomposition (redundancy, synergy, uniqueness) to a Stable Diffusion 2.1 denoiser at pixel/image level to interpret how prompt tokens shape generated images, with a conditional variant CPID.
Problem
Text-to-image diffusion models are opaque: it is unclear how individual prompt words and their interactions drive generated images, and why they fail on homonyms, synonyms, co-hyponyms, and show biases. Prior pixel-level MI methods only give coarse, individual-token information.
Approach
Extends pixel-wise mutual information estimated from denoiser MMSE differences to a partial information decomposition of two prompt phrases, using BERT-based phrase probabilities for redundancy, plus a conditional variant (CPID) that conditions on the rest of the prompt.
Key results
- Redundancy for male gender is higher than female for male-stereotyped jobs (e.g., Doctor 0.871 vs 0.090), and average female redundancy (0.194) is lower than male (0.286).
- For ethnicity, average redundancy is lowest for Black (0.133) versus Asian (0.233), Caucasian (0.255), and Hispanic (0.236), suggesting bias against Black generations.
- Qualitatively, PID redundancy/synergy maps localize synonym and co-hyponym regions and prompt-intervention removals of redundant words change images little, where MI, CMI, and DAAM maps are sparse or misleading.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use male rather than female gender in occupation prompts | Stable Diffusion 2.1 · Gender Bias | redundancy: Male 0.286; Female 0.194 | Mixed |
| Pair male gender with male-stereotyped occupations | Stable Diffusion 2.1 · Gender Bias | redundancy: Doctor: 0.871 vs 0.090; Plumber: 0.605 vs 0.038 | Helped |
| Pair female gender with female-stereotyped occupations | Stable Diffusion 2.1 · Gender Bias | redundancy: Babysitter: 0.531 vs 0.240; Teacher: 0.419 vs 0.098 | Helped |
| Pair occupations with different ethnicity prompts | Stable Diffusion 2.1 · Ethnicity Bias | redundancy: Athlete: Black 0.321; Engineer: Asian 0.156; Butcher: Caucasian 0.474; Nurse: Hispanic 0.219 | Mixed |
| Use Black ethnicity prompts across occupations | Stable Diffusion 2.1 · Ethnicity Bias | redundancy: 0.133 | Hurt |
| Use CPID instead of PID to account for the rest of the prompt | DiffusionPID · Complex prompts | localization quality | Helped |
| Use PID redundancy maps rather than MI, CMI, or DAAM maps to identify overlapping synonym regions | Stable Diffusion 2.1 · Synonyms | localization quality | Helped |
| Remove a redundant word identified by PID from the prompt | Stable Diffusion 2.1 · Prompt intervention | image change | No clear effect |
Surprising findings
- Average female redundancy across occupations is very low, which the authors interpret as bias against generating females for any occupation.
- MI, CMI, and DAAM maps are highly activated for words whose removal barely changes the image, so they mislead on redundant-term identification.
Limitations
- Only two phrases are decomposed at a time; extension to more variables is not done.
- Requires access to the diffusion model's denoiser, so closed-source models cannot be analyzed.
- Estimator assumes an optimal denoiser and has no guaranteed bounds; estimates depend on Monte Carlo sampling noise.
Experimental setup
Datasets: COCO WordNet
Benchmarks: gender bias occupation prompts (188 occupations, 376 prompts) ethnicity bias occupation prompts (752 prompts) homonym prompts (242) synonym prompts (132) COCO co-hyponym prompts (197 pairs) WordNet co-hyponym prompts (798 pairs)
Metrics: normalized image-level redundancy synergy maps uniqueness maps qualitative map inspection
Baselines: MI CMI DAAM
Builds on: partial information decomposition pixel-wise mutual information for diffusion models DAAM BERT
Closest papers by meaning
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.92)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.91)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.89)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.