Research map / Image generation / Text-to-image diffusion
AID: Attention Interpolation of Text-to-Image Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AID, a training-free method that interpolates text-to-image diffusion conditions by fusing inner/outer interpolated cross- and self-attention and selecting coefficients with a Beta prior, plus PAID for prompt-guided paths.
Problem
Interpolating between two text conditions in text-to-image diffusion models yields sequences with poor consistency, smoothness, and fidelity, since linear text-embedding interpolation ignores spatial layout constraints from self-attention.
Approach
Interpolate keys and values of both cross-attention and self-attention (fused with source keys/values), use Beta-distribution-selected interpolation coefficients for smoother transitions, and optionally inject a guidance prompt into the fused attention to control the path; all without training.
Key results
- AID-O reduces consistency (LPIPS) from 0.3645 to 0.2905 on CIFAR-10 and FID fidelity from 118.05 to 51.43 versus text embedding interpolation.
- AID-I achieves the best smoothness on LAION-Aesthetics (0.8152 vs 0.7424 for TEI).
- Human preference study (320 trials on Mechanical Turk) predominantly favors AID-I or AID-O over TEI, e.g. TEI preferred in only 0-8.75% of trials per category.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AID-O instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · CIFAR-10 | consistency: 0.3645 -> 0.2905 | Helped |
| Use AID-O instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · CIFAR-10 | fidelity (FID): 118.05 -> 51.43 | Helped |
| Use AID-I instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · CIFAR-10 | smoothness: 0.7531 -> 0.7861 | Helped |
| Use AID-O instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · LAION-Aesthetics | consistency: 0.3867 -> 0.2944 | Helped |
| Use AID-O instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · LAION-Aesthetics | fidelity (FID): 142.38 -> 82.01 | Helped |
| Use AID-I instead of TEI | Conditional interpolation with Stable Diffusion 1.4 · LAION-Aesthetics | smoothness: 0.7424 -> 0.8152 | Helped |
| Add attention interpolation (ablation) | AID-O on CIFAR-10 · CIFAR-10 | consistency: 0.3645 -> 0.3201 | Helped |
| Add Beta prior (ablation) | AID-O on CIFAR-10 · CIFAR-10 | smoothness: 0.7846 -> 0.8517 | Helped |
| Add self-attention fusion (ablation) | AID-O on CIFAR-10 · CIFAR-10 | fidelity (FID): 155.01 -> 51.43 | Helped |
| Add Beta prior (ablation) | AID-O on CIFAR-10 · CIFAR-10 | smoothness: 0.6236 -> 0.7831 | Helped |
| Remove Beta prior (ablation) | AID-O on CIFAR-10 · CIFAR-10 | consistency: 0.2905 -> 0.2411 | Helped |
| Add self-attention fusion without Beta prior (ablation) | AID-O on CIFAR-10 · CIFAR-10 | smoothness: 0.7846 -> 0.6236 | Hurt |
Surprising findings
- Interpolating text embeddings is mathematically equivalent to interpolating cross-attention keys and values, yet self-attention keys and values exert a stronger spatial-layout influence than cross-attention.
- Attention interpolation without self-attention fusion gives poor fidelity (FID 101-155) and removing fusion from self-attention yields the best consistency but lowest smoothness (0.6236).
Limitations
- Method is post-hoc and performance depends on the capability of the base text-to-image diffusion model.
- AID-I has poorer fidelity than AID-O and worse than denoising interpolation on quantitative FID.
- Smoothness/consistency are measured on discrete sequences with LPIPS and Gini, which may not fully capture perceived quality.
Experimental setup
Datasets: CIFAR-10 LAION-Aesthetics
Benchmarks: CIFAR-10 LAION-Aesthetics
Metrics: LPIPS-based consistency Gini-based smoothness FID fidelity CLIP score human preference smoothness for editing
Baselines: Text Embedding Interpolation Denoising Interpolation Compositional Energy-based Model (CEBM) RRR Prompt-to-Prompt EDICT
Builds on: Prompt-to-Prompt EDICT DDIM Stable Diffusion
Closest papers by meaning
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.89)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models (ICLR 2024 · similarity 0.88)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.88)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.88)
- Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.