Research map / Image generation / Text-to-image diffusion
DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEADiff, an encoder-based style transfer method for Stable Diffusion that decouples style and semantics via two instructed Q-Formers, injects them into mutually exclusive cross-attention layers, and trains non-reconstructively on paired synthetic images.
Problem
Encoder-based reference-style transfer for text-to-image diffusion models significantly impairs text controllability because the extracted image features couple style with semantics and reconstruction training lets the encoder dominate over the text condition.
Approach
A dual decoupling representation extraction mechanism uses two Q-Formers instructed by "style" and "content" text to extract disentangled features, trained with non-reconstructive paired-image objectives, and a disentangled conditioning mechanism injects style features only into fine cross-attention layers and content features into coarse layers.
Key results
- DEADiff reaches 0.289 text alignment in the ablation versus 0.148 for the baseline reconstruction model, while keeping style similarity at 0.224.
- In the main comparison DEADiff achieves the highest image quality (5.840) and text alignment 0.284, with 69.0 subjective preference score.
- Style similarity of DEADiff (0.229) is above T2I-Adapter (0.241 is higher) but DEADiff trades slight style similarity for much better text alignment than T2I-Adapter (0.224).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add disentangled conditioning mechanism (DCM) (ablation) | DEADiff baseline · Evaluation set | text alignment: 0.148 -> 0.224 | Helped |
| Add disentangled conditioning mechanism (DCM) (ablation) | DEADiff baseline · Evaluation set | style similarity: 0.274 -> 0.259 | Hurt |
| Add non-reconstructive style representation extraction (STRE) (ablation) | DEADiff with DCM · Evaluation set | text alignment: 0.224 -> 0.286 | Helped |
| Add non-reconstructive semantic representation extraction (SERE) (ablation) | DEADiff with DCM and STRE · Evaluation set | text alignment: 0.286 -> 0.287 | Helped |
| Add reconstruction task combining style and content representations (ablation) | DEADiff with DCM, STRE, and SERE · Evaluation set | text alignment: 0.287 -> 0.289 | Helped |
Surprising findings
- Ablation shows the reconstruction-trained baseline achieves the highest style similarity (0.274) but very poor text alignment (0.148), illustrating the controllability trade-off.
Limitations
- Slightly inferior color accuracy compared with optimization-based StyleDrop
- Style similarity metric is self-proposed since no precise metric exists
- Training data is proprietary, generated by Midjourney
Experimental setup
Datasets: self-constructed Midjourney image-text pairs (1.06M) self-constructed same-style pairs (160k) self-constructed same-semantics pairs (1.06M) WikiArt Civitai
Benchmarks: custom 32-style evaluation set with 35 prompts (WikiArt, Civitai)
Metrics: style similarity text alignment (CLIP) image quality user study subjective preference
Baselines: InST CAST StyTr2 T2I-Adapter IP-Adapter StyleAdapter StyleDrop
Builds on: Stable Diffusion v1.5 BLIP-Diffusion IP-Adapter Q-Former
Closest papers by meaning
- Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models (ICLR 2025 · similarity 0.87)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.87)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.87)
- OmniStyle: Filtering High Quality Style Transfer Data at Scale (CVPR 2025 · similarity 0.87)
- Z*: Zero-shot Style Transfer via Attention Reweighting (CVPR 2024 · similarity 0.86)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.