Research map / Image generation / Text-to-image diffusion
The Hidden Language of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Conceptor decomposes a textual concept's representation in Stable Diffusion into a sparse weighted combination of interpretable vocabulary tokens learned by an MLP trained with a denoising objective.
Problem
Text-to-image diffusion models learn internal concept representations that are opaque; existing concept-based interpretability methods (CNN-based, prompt tuning, captioning) do not produce faithful, meaningful decompositions for generative diffusion models.
Approach
Learn a pseudo-token as a linear combination of top vocabulary token embeddings, with coefficients produced by a 2-layer MLP applied to each word embedding, trained with the diffusion denoising loss on concept images plus a sparsity loss that keeps the top-50 tokens representative; per-image decomposition is done by iterative token removal checked with CLIP similarity.
Key results
- Conceptor achieves CLIP pairwise 86.2 and LPIPS 0.44 versus 83.0 and 0.53 for the best baseline (PCA), and FID entire set 9.8 versus 18.2 for PEZ.
- In a user study over 160 responses, 70% preferred Conceptor's decomposition over BLIP-2 captions and PEZ.
- Across training-set and seed changes, 80.3% of top-10 tokens and 70.7% of top-25 tokens for concrete concepts were preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the MLP with directly learned coefficient weights (ablation) | Conceptor · Professions | CLIP pairwise: 87.0 ± 5.5 -> 78.0 ± 6.7 | Hurt |
| Remove the sparsity loss (ablation) | Conceptor · Professions | CLIP pairwise: 87.0 ± 5.5 -> 80.3 ± 11.6 | Hurt |
| Reduce the number of decomposition tokens to 10 (ablation) | Conceptor · Professions | CLIP pairwise: 87.0 ± 5.5 -> 82.9 ± 7.8 | Hurt |
| Increase the number of decomposition tokens to 100 (ablation) | Conceptor · Professions | CLIP pairwise: 87.0 ± 5.5 -> 85.6 ± 6.9 | No clear effect |
| Select the top 50 tokens by CLIP similarity to the mean image (ablation) | Conceptor · Professions | CLIP pairwise: 87.0 ± 5.5 -> 80.1 ± 9.9 | Hurt |
Surprising findings
- The painter concept relies on renowned artist names not in the prompt: 67 of 100 painter images had at least one famous artist in their decomposition.
- Some concepts are represented by exemplars, e.g., removing Trump from a president image yields an Obama-like president, showing interpolation of identities.
Limitations
- Decomposition elements are restricted to single vocabulary tokens, so multi-word phrases cannot appear as single elements.
- Token impact depends on context within the decomposition, so single-token meaning can differ from its effect in the mix.
- Exact-match token consistency is lower for abstract concepts and for deeper token ranks (57.9% of top-50 for concrete concepts).
Experimental setup
Datasets: CIFAR-10 classes Bias in Bios professions ConceptNet concepts Stable Diffusion-generated concept images Best 30 Stable Diffusion Prompts website prompts
Benchmarks: 188-concept diffusion interpretability suite (CIFAR-10, Bias in Bios, emotions, actions, complex prompts, ConceptNet)
Metrics: CLIP pairwise similarity LPIPS FID per concept FID entire set token diversity (SentenceBERT) user study preference
Baselines: PEZ BLIP-2 sentence BLIP-2 token PCA NMF k-means
Builds on: Textual Inversion PEZ TCAV CRAFT
Closest papers by meaning
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.91)
- Unveiling Concept Attribution in Diffusion Models (NeurIPS 2025 · similarity 0.91)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.91)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.90)
- Non-confusing Generation of Customized Concepts in Diffusion Models (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.