Research map / Image generation / Editing and style transfer
Free-Lunch Color-Texture Disentanglement for Stylized Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SADis, a tuning-free method that disentangles color and texture from separate CLIP image-prompt embeddings (via image-prompt additivity, SVD rescaling) and applies a noise-regularized whitening-coloring transform to stylize SDXL generations.
Problem
Stylized text-to-image generation transfers color and texture from reference images entangled together, offering no independent control over color and texture attributes.
Approach
Exploits an image-prompt additivity property in the CLIP image embedding space: color embeddings are obtained by subtracting grayscale embeddings from color ones, texture embeddings via grayscale plus SVD singular-value reweighting, and a noise-regularized whitening-coloring transform on latents aligns colors while preserving texture.
Key results
- SADis achieves MS-SWD 5.57 and C-Hist 0.96 versus 9.51 and 1.20 for the best baseline SDXL, a clear color-alignment gain.
- In the user study, SADis is preferred in 37.84% of 'both' comparisons versus 14.55% for the next best method (IP-Adapter).
- Removing the WCT module raises MS-SWD from 5.57 to 8.05, showing the color gain comes mainly from the regularized WCT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SADis instead of the strongest baseline | SDXL stylization · WikiArt and StyleDrop | MS-SWD: 9.51 -> 5.57 | Helped |
| Use SADis instead of the strongest baseline | SDXL stylization · DREAMBENCH++ complex prompts | SWD: 9.00 -> 6.08 | Helped |
| Use SADis instead of the strongest baseline | SDXL stylization · WikiArt and StyleDrop | CLIP-I: 0.84 -> 0.74 | Hurt |
| add SVD rescaling (ablation) | SADis · WikiArt and StyleDrop | MS-SWD: 5.70 -> 5.57 | Helped |
| add SVD rescaling (ablation) | SADis · WikiArt and StyleDrop | CLIP-I: 0.76 -> 0.74 | Hurt |
| add regularized WCT (ablation) | SADis · WikiArt and StyleDrop | MS-SWD: 8.05 -> 5.57 | Helped |
| add regularized WCT (ablation) | SADis · WikiArt and StyleDrop | CLIP-I: 0.75 -> 0.74 | Hurt |
| increase RegWCT scale from 0 to 1.0 (ablation) | SADis · WikiArt and StyleDrop | MS-SWD: 8.05 -> {5.18} | Helped |
| increase RegWCT scale from 0 to 1.0 (ablation) | SADis · WikiArt and StyleDrop | CLIP-I: 0.747 -> {0.740} | Hurt |
| increase SVD scaling factor gamma from 0 to 0.011 (ablation) | SADis · WikiArt and StyleDrop | MS-SWD: 5.70 -> 5.20 | Helped |
| increase SVD scaling factor gamma from 0 to 0.011 (ablation) | SADis · WikiArt and StyleDrop | CLIP-I: 0.759 -> 0.713 | Hurt |
| increase beta from 0.5 to 1.5 (ablation) | SADis · WikiArt and StyleDrop | MS-SWD: 5.20 -> 6.17 | Hurt |
Surprising findings
- IP-Adapter achieves the highest texture scores but suffers severe semantic leakage from the texture reference, lowering text-image alignment (CLIP 0.233 vs 0.281 for SADis).
Limitations
- Content-consistent stylization is not well handled; the authors note this as future work.
- Identity and object information can be diminished under image-prompt additivity manipulations.
- Relies on hand-tuned hyperparameters (lambda, omega, gamma, beta, timestep interval) chosen by ablation.
Experimental setup
Datasets: WikiArt StyleDrop DreamBench++
Benchmarks: WikiArt StyleDrop DreamBench++
Metrics: CLIP score MS-SWD C-Hist GPT-4o color score CLIP-I KID user study
Baselines: SDXL IP-Adapter InstantStyle Artist DEADiff StyleDrop DreamStyler CSGO
Builds on: IP-Adapter SDXL Whitening-Coloring Transform InstantStyle ControlNet
Closest papers by meaning
- Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation (NeurIPS 2026 · similarity 0.88)
- Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation (CVPR 2025 · similarity 0.88)
- SigStyle: Signature Style Transfer via Personalized Text-to-Image Models (AAAI 2025 · similarity 0.87)
- StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer (CVPR 2025 · similarity 0.87)
- Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer (CVPR 2024 · similarity 0.87)
- CSGO: Content-Style Composition in Text-to-Image Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.