Research map / Image generation / Editing and style transfer
Z*: Zero-shot Style Transfer via Attention Reweighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Z* performs zero-shot, training-free image style transfer by using dual DDIM-inverted denoising paths in Stable Diffusion v1.5 and a cross-attention reweighting that fuses style and content keys/values inside the softmax.
Problem
Image style transfer usually requires training networks with style losses, and existing diffusion-based style transfer methods need per-style embedding training while often losing content structure or style fidelity.
Approach
Vanilla latent diffusion models can extract style directly from a style image via dual denoising paths (DDIM inversion of content and style images) and attention injection; a reweighting that places the scaling factor λ inside the softmax (rather than naive additive blending) balances content preservation and style transfer without retraining.
Key results
- In a user study with 55 participants and 1,760 votes, Z* was preferred over most baselines on content preservation; CAST and StyTr2 slightly outperformed it on style (differences described as insignificant).
- Z* uses Stable Diffusion v1.5 with 30 denoising steps, injecting attention in layers 10-15 from steps 5-30, and fixed λ=1.2 chosen by ablation.
- Naive attention blending and simple addition were shown qualitatively to lose content structure or over-emphasize content compared with the reweighted attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Z* instead of the compared style-transfer methods | Z* · User study | content preservation preference | Helped |
| Use Z* instead of the compared style-transfer methods | Z* · User study | overall preference | Helped |
| Use Z* instead of CAST or StyTr2 | Z* · User study | style representation preference | Hurt |
| Start attention injection later in denoising (ablation) | Z* · Style-transfer examples | content preservation | Helped |
| Start attention injection later in denoising (ablation) | Z* · Style-transfer examples | style patterns | Hurt |
| Increase the overall number of denoising steps with attention injection (ablation) | Z* · Style-transfer examples | style patterns | Helped |
| Inject attention into all U-Net layers rather than selected decoder layers (ablation) | Z* · Style-transfer examples | content structure | Hurt |
| Inject attention into low-resolution U-Net layers (layers 5-10) (ablation) | Z* · Style-transfer examples | style patterns | Hurt |
| Inject attention into encoder layers (layers 0-5) (ablation) | Z* · Style-transfer examples | content structure | Hurt |
| Remove the content self-attention component (naive setting) (ablation) | Z* · Style-transfer examples | content preservation | Hurt |
| Replace cross-attention reweighting with simple addition of content self-attention (ablation) | Z* · Style-transfer examples | content details | Hurt |
| Use cross-attention reweighting instead of naive setting or simple addition (ablation) | Z* · Style-transfer examples | content-style balance | Helped |
Surprising findings
- Vanilla diffusion models can extract style from a style image alone, without text prompts or per-style training.
Limitations
- Evaluation relies mainly on qualitative comparisons and a user study; no standard quantitative style metrics are reported.
- One-to-many style transfer with N style images is left for future work due to memory cost.
- Style-only results are slightly behind CAST and StyTr2 in the user study.
Experimental setup
Metrics: user preference votes (content, style, overall)
Baselines: ArtFlow AdaAttN IEST StyTr2 CAST QuanArt InST VCT StyleDiff
Builds on: Stable Diffusion DDIM inversion Plug-and-Play MasaCtrl Prompt-to-prompt
Closest papers by meaning
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.93)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.90)
- Attention Distillation: A Unified Approach to Visual Characteristics Transfer (CVPR 2025 · similarity 0.90)
- ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank (AAAI 2024 · similarity 0.90)
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.89)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.