Research map / Image generation / Fast diffusion sampling
ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConceptAttention, a training-free method that repurposes multi-modal DiT attention layers (Flux, SD3.5) to produce concept embeddings whose linear projections onto image output vectors yield sharp saliency maps for zero-shot segmentation.
Problem
Interpreting text-to-image diffusion transformers: existing saliency methods rely on UNet cross-attention, which in multi-modal DiTs is limited to prompt tokens and cannot handle open-set textual concepts.
Approach
Generate contextualized concept embeddings by running concept tokens through the prompt projection matrices of each MM-DiT attention layer, with concepts attending to image tokens one-way (without influencing the image); then compute saliency as a softmax dot product between image and concept vectors in the attention output space, which beats cross-attention maps.
Key results
- On ImageNet-Segmentation, ConceptAttention on Flux reaches 83.07 Acc, 71.04 mIoU, 90.45 mAP, versus 74.92 mIoU for Flux raw cross-attention.
- On PascalVOC single class, ConceptAttention (Flux) achieves 76.45 mIoU, outperforming TransInterp (57.08) and the best DINO baseline (64.33).
- On multi-class PascalVOC, ConceptAttention reaches 86.99 Acc and 51.39 mIoU, beating TextSpan, DAAM and Flux cross-attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConceptAttention rather than Flux cross-attention maps | Flux-Schnell · ImageNet-Segmentation | mIoU: 59.90 -> 71.04 | Helped |
| Use ConceptAttention rather than DINO self-attention | ImageNet-Segmentation · ImageNet-Segmentation | mIoU: 69.44 -> 71.04 | Helped |
| Use ConceptAttention rather than Flux cross-attention maps | Flux-Schnell · PascalVOC (Single Class) | mIoU: 54.77 -> 76.45 | Helped |
| Use ConceptAttention rather than Flux cross-attention maps | Flux-Schnell · PascalVOC (Multi Class) | mIoU: 27.04 -> 51.39 | Helped |
| Use attention output space instead of cross-attention space (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 59.90 -> 71.04 | Helped |
| Add pixelwise softmax to output-space maps (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 64.95 -> 71.04 | Helped |
| Add pixelwise softmax to cross-attention maps (ablation) | ConceptAttention cross-attention variant · ImageNet-Segmentation | mIoU: 49.91 -> 59.90 | Helped |
| Add pixelwise softmax to value-space maps (ablation) | ConceptAttention value-space variant · ImageNet-Segmentation | mIoU: 29.81 -> 29.68 | Hurt |
| Use output-space maps instead of value-space maps (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 29.68 -> 71.04 | Helped |
| Add both cross-attention and self-attention among concepts (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 61.96 -> 71.04 | Helped |
| Add cross-attention between concepts and image tokens (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 34.85 -> 71.04 | Helped |
| Add both cross-attention and self-attention among concepts (ablation) | ConceptAttention · ImageNet-Segmentation | mIoU: 35.72 -> 71.04 | Helped |
Surprising findings
- Middle diffusion timesteps (around 500 of 1000) give better segmentation than the least-noised late timesteps.
- Projecting onto the value space performs far worse than cross-attention maps, while the attention output space performs best.
Limitations
- Struggles to distinguish very similar concepts, e.g. capturing a halo around the sun instead of its boundary.
- When no relevant concept is present, it selects the most similar concept even if incorrect.
- Evaluation on video is qualitative only.
Experimental setup
Datasets: ImageNet-Segmentation PascalVOC 2012
Benchmarks: ImageNet-Segmentation PascalVOC 2012 (single class) PascalVOC 2012 (multi-class)
Metrics: Acc mIoU mAP
Baselines: LRP Partial-LRP Rollout ViT Attention GradCAM TextSpan TransInterp CLIPasRNN OVAM DAAM DINO SA DINOv2 SA
Builds on: Flux Stable Diffusion 3 DAAM rectified flow
Closest papers by meaning
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.89)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
- DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.