Research map / Image generation / Text-to-image diffusion
$\text{I}^2\text{AM}$: Interpreting Image-to-Image Latent Diffusion Models via Bi-Attribution Maps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes I²AM, bidirectional cross-attention attribution maps (reference-to-generated and generated-to-reference) aggregated over time, heads and layers to interpret image-to-image latent diffusion models, plus the IMACS metric for inpainting.
Problem
Interpretability of image-to-image latent diffusion models is underexplored: existing cross-attention attribution methods are designed for text-to-image models with discrete tokens and only capture one direction of influence.
Approach
Treat reference and generated image patches as both queries and keys in cross-attention, compute attention in both directions, and aggregate across diffusion time steps, heads and layers (unified, time, head, layer and specific-reference maps); also introduce IMACS, a mask-alignment score for reference-based inpainting.
Key results
- On VITON-HD, IMACS_g correlates with downstream metrics: StableVITON scores 0.3083 IMACS_g and 10.68 FID versus DCI-VTON's 0.0785 IMACS_g and 13.10 FID.
- A debugged custom model with added attention-alignment losses improves IMACS_g from 0.0833 to 0.2215 and IMACS_r from 0.0403 to 0.0948, with FID reduced to 11.54 from 11.66.
- On COCO object detection with Paint-by-Example outputs, I²AM reaches mIoU 0.2416 and outperforms YOLOv3 in some cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DCML, TV, and CWG attention-alignment losses to the custom model (ablation) | Custom VITON model · VITON-HD | IMACS_g: 0.0833 -> 0.2215 | Helped |
| Add DCML, TV, and CWG attention-alignment losses to the custom model (ablation) | Custom VITON model · VITON-HD | SSIM: 0.8396 -> 0.8644 | Helped |
| Set the attribution-map threshold δ to 0.4 rather than 0.2 or 0.3 (ablation) | I²AM on PBE · COCO PBE-generated images | mIoU_gen>0.5: 0.2413 -> 0.2416 | Mixed |
| Set the attribution-map threshold δ to 0.6 rather than 0.4 (ablation) | I²AM on PBE · COCO PBE-generated images | mIoU_gen>0.5: 0.2416 -> 0.24 | Hurt |
| Increase the IMACS penalty factor λ (ablation) | IMACS evaluation · VITON-HD | IMACS_g: DCI-VTON: 0.4291 -> 0.0785 -> -0.3051 -> -1.2155; StableVITON: 0.5376 -> 0.3083 -> 0.07 -> -0.1219 | Hurt |
| Use I²AM’s reversed softmax direction instead of DAAM’s direction for I2I attribution maps | I²AM · I2I attribution visualization | interpretability | Helped |
| Use classifier-free guidance | Paint-by-Example · Image inpainting | attention alignment with inpainting mask | Helped |
Surprising findings
- Applying DAAM's softmax direction to I2I models yields uniform, uninterpretable attention maps, so the softmax direction must be reversed for reference-to-generated analysis.
- Without classifier-free guidance, Paint-by-Example attention scores spill beyond the inpainting mask region.
Limitations
- Primarily tested in paired reference-based settings; unpaired scenarios only preliminarily indicated.
- Generated-to-reference maps are often uninformative at the layer level and require the specific-reference map (SRAM) instead.
- Evaluation of IMACS relies on a small number of models (four), and the custom model is the authors' own.
Experimental setup
Datasets: VITON-HD OpenImages COCO
Benchmarks: COCO VITON-HD
Metrics: mIoU FID KID LPIPS SSIM IMACS
Baselines: DAAM YOLOv3 DCI-VTON StableVITON
Builds on: DAAM Stable Diffusion Latent diffusion models DDIM
Closest papers by meaning
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.89)
- Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects (CVPR 2025 · similarity 0.88)
- Exploring Local Memorization in Diffusion Models via Bright Ending Attention (ICLR 2025 · similarity 0.88)
- AID: Attention Interpolation of Text-to-Image Diffusion (NeurIPS 2024 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.