Research map / Image generation / Editing and style transfer
Attention Distillation: A Unified Approach to Visual Characteristics Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an attention distillation loss on self-attention features of pretrained Stable Diffusion, used both for latent optimization and as an improved classifier guidance, to transfer style, appearance and texture from a reference image.
Problem
Existing plug-and-play KV-injection methods for transferring style, appearance, or texture from a reference image to generated images give insufficient stylization due to domain gap, error accumulation, and architectural limits of the residual branch.
Approach
Define an L1 attention distillation loss between the self-attention output of the target (queries from target, keys/values from reference) and the current target self-attention output, and optimize the latent by backpropagation (or inject it as an Adam-managed classifier guidance during DDIM sampling), so gradients also flow through the identity connection.
Key results
- Attention distillation loss yields visually superior style and texture transfer versus KV-injection in qualitative comparisons (Fig. 3).
- User study with 1500 responses from 50 participants shows the method is preferred by significant margins over baselines across three transfer tasks.
- Adam-managed guidance strength makes sampling-time guidance work with a universal learning rate, where manual guidance strength often loses reference textures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Adam optimizer to manage guidance strength instead of manually setting it (ablation) | Attention Distillation guided sampling · style and appearance transfer | visual match to the reference | Helped |
| Increase optimization iterations per timestep from 1 to 2 (ablation) | Attention Distillation guided sampling · text-to-image generation | generated result quality: 10s -> 19s | Helped |
| Increase optimization iterations per timestep from 2 to 3 (ablation) | Attention Distillation guided sampling · text-to-image generation | generated result quality: 19s -> 28s | Helped |
| Vary the content loss weight λ (ablation) | Content-preserving optimization · style transfer and appearance transfer | content preservation | Mixed |
| Fine-tune the VAE decoder on the reference image (ablation) | Attention Distillation · image synthesis | reconstruction quality | Helped |
Surprising findings
- KV-injection with 100 sampling steps loses reference style details, while optimizing the same latent with the AD loss recovers them, attributed to gradient flow through the identity connection.
Limitations
- Strength of guidance is tricky to configure without an Adam optimizer; manual settings often fail.
- Optimization-based synthesis requires multiple iterations per step, adding computation time.
Experimental setup
Metrics: user preference score
Baselines: CSGO StyleShot StyleID StyTR2 NST Cross-Image Attention SpliceViT Visual Style Prompting InstantStyle RB-Modulation GCD Self-Rectification
Builds on: KV-injection classifier guidance MultiDiffusion Cross-Image Attention ControlNet
Closest papers by meaning
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.90)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.89)
- DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling (NeurIPS 2025 · similarity 0.88)
- Diffusion Self-Distillation for Zero-Shot Customized Image Generation (CVPR 2025 · similarity 0.88)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.88)
- CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.