Research map / Image generation / Editing and style transfer
Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free artistic style transfer that injects style key/value into the self-attention layers of Stable Diffusion 1.4, with query preservation, attention temperature scaling, and initial latent AdaIN.
Problem
Diffusion-based style transfer either needs per-style inference-time optimization (fine-tuning or textual inversion) or cannot leverage large-scale latent diffusion models such as Stable Diffusion.
Approach
Substitute the key and value of content self-attention with those of the style image during DDIM reverse sampling (in the later decoder layers), blend content and stylized queries to preserve structure, sharpen attention with a temperature factor, and align initial-noise channel statistics with AdaIN for color transfer.
Key results
- ArtFID of 28.801 vs 31.420 for the next-best conventional baseline (AesPA-Net), and FID 18.131 (lowest among all methods).
- Inference of 12.4 s per pair on a TITAN RTX versus 355.9 s for DiffStyle and about 800 s for DiffuseIT and InST.
- Ablation: removing style injection raises ArtFID from 28.80 to 43.72; removing initial latent AdaIN raises FID from 18.13 to 20.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove style injection (ablation) | Style Injection · Style transfer | ArtFID: 28.80 -> 43.72 | Hurt |
| remove attention temperature scaling (ablation) | Style Injection · Style transfer | LPIPS: 0.505 -> 0.542 | Hurt |
| remove initial latent AdaIN (ablation) | Style Injection · Style transfer | FID: 18.13 -> 20.05 | Hurt |
| set query preservation ratio γ to 0.6 instead of 0.75 (ablation) | Style Injection · Style transfer | ArtFID: 28.80 -> 27.97 | Helped |
| set query preservation ratio γ to 0.6 instead of 0.75 (ablation) | Style Injection · Style transfer | LPIPS: 0.505 -> 0.535 | Hurt |
| use attention temperature scaling (τ = 1.5) instead of no scaling (ablation) | Style Injection · Style transfer | FID: 17.81 -> 18.13 | Helped |
| use initial latent AdaIN (ablation) | Style Injection · Style transfer | LPIPS: 0.390 -> 0.505 | Mixed |
Surprising findings
- Injecting query, key and value of style into self-attention still leaves the content's color tone largely unchanged, so color transfer must come from the initial latent statistics.
Limitations
- Relies on Stable Diffusion 1.4 with empirically chosen hyperparameters (gamma=0.75, tau=1.5).
- Evaluation uses a small set of 800 stylized images and ArtFID, which depends on LPIPS and FID rather than human judgment.
Experimental setup
Datasets: MS-COCO WikiArt
Benchmarks: MS-COCO content / WikiArt style evaluation (800 stylized images)
Metrics: ArtFID FID LPIPS CFSD inference time
Baselines: AesPA-Net CAST StyTR2 EFDM MAST AdaAttN ArtFlow AdaConv AdaIN DiffuseIT InST DiffStyle
Builds on: Prompt-to-Prompt Plug-and-Play DDIM inversion Latent Diffusion Model AdaIN
Closest papers by meaning
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.93)
- ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank (AAAI 2024 · similarity 0.91)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.90)
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.90)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.89)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.