Research map / Image generation / Editing and style transfer
Color Conditional Generation with Sliced Wasserstein Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SW-Guidance, a training-free diffusion sampling method that steers generation toward a reference image's RGB color distribution by minimizing a differentiable sliced 1-Wasserstein distance.
Problem
Color-conditional text-to-image generation: producing images that match a reference color palette while keeping semantic fidelity to the text prompt, without transferring other style features.
Approach
Adds a gradient of a differentiable Sliced Wasserstein distance between the color distribution of the decoded predicted clean image and the reference palette to each diffusion step, optimizing a control shift u on the latent before DDIM denoising (universal-guidance style), and compares with color-transfer and stylized-generation baselines.
Key results
- On SDXL, SW-Guidance achieves 2-Wasserstein distance 0.0297 vs 0.0543 for the best baseline (hm-mkl-hm) and 0.3824 unconditional.
- SW-Guidance has the highest CLIP-T (0.270 on SDXL is not highest; among stylized methods it reports the highest CLIP-T of 0.270 vs 0.238-0.214 for IP-Adapter/RB-Modulation/InstantStyle as stated in text), with CLIP-IQA 0.285 second only to... as reported in table.
- On SD-1.5, SW-Guidance reaches 2-Wasserstein 0.0328 vs 0.0572 for hm-mkl-hm; generation takes about 30 s on SD-1.5 and about 1 minute on SDXL on an RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SW-Guidance instead of the strongest color-transfer baseline | SDXL · ContraStyles test set | 2-Wasserstein distance: 0.0543 ± 0.0011 -> 0.0297 ± 0.0005 | Helped |
| apply SW-Guidance instead of the strongest color-transfer baseline | SD1.5 · ContraStyles test set | 2-Wasserstein distance: 0.0572 ± 0.0011 -> 0.0328 ± 0.0003 | Helped |
| use standard SW rather than Energy-Based SW (ablation) | SW-Guidance · ContraStyles test set, SD1.5 ablation | 2-Wasserstein distance: 0.0385 ± 0.0006 vs 0.0390 ± 0.0006 | No clear effect |
| use standard SW rather than Distributional SW (ablation) | SW-Guidance · ContraStyles test set, SD1.5 ablation | 2-Wasserstein distance: 0.0547 ± 0.0006 -> 0.0385 ± 0.0006 | Helped |
| use standard SW rather than Generalized SW (ablation) | SW-Guidance · ContraStyles test set, SD1.5 ablation | 2-Wasserstein distance: 0.0879 ± 0.0014 -> 0.0385 ± 0.0006 | Helped |
| use SW rather than mean and covariance matching (ablation) | SW-Guidance · ContraStyles test set, SD1.5 ablation | 2-Wasserstein distance: 0.1064 ± 0.0013 -> 0.0385 ± 0.0006 | Helped |
| add mean and covariance terms to SW (ablation) | SW-Guidance · ContraStyles test set, SDXL ablation | 2-Wasserstein distance: 0.0297 ± 0.0005 -> 0.0305 ± 0.0006 | Hurt |
| use SW alongside mean and covariance terms instead of moments alone (ablation) | SW-Guidance · ContraStyles test set, SDXL ablation | 2-Wasserstein distance: 0.1176 ± 0.0016 -> 0.0297 ± 0.0005 | Helped |
| apply SW-Guidance instead of RB-Modulation | SDXL generation · image generation | generation time: 2 minutes -> around 1 minute | Helped |
| use SW-Guidance instead of mean/covariance matching only | SDXL · content diversity evaluation | FID: 53.16 -> 58.40 | Hurt |
Surprising findings
- Adding mean and covariance matching terms to the SW loss did not improve color similarity and reduced image quality; moments-only guidance was insufficient.
- Energy-based, distributional and generalized SW variants showed no substantial content-score advantage over plain SW for 3D color transfer.
Limitations
- Sensitive to conflicting color words in text prompts, which cause visual artifacts.
- Combining with attention-based stylizers (e.g., InstantStyle) can produce clashing color guidance.
- Requires backpropagation through the U-Net, which increases compute.
Experimental setup
Datasets: ContraStyles Unsplash Lite
Benchmarks: ContraStyles (first 1000 prompts)
Metrics: 2-Wasserstein distance (RGB) CLIP-IQA CLIP-T FID
Baselines: Histogram matching MKL CT WCT2 PhotoNAS PhotoWCT2 ModFlows IP-Adapter InstantStyle RB-Modulation
Builds on: Universal Diffusion Guidance Diffusion Posterior Sampling Sliced Wasserstein distance DDIM Distributional Sliced Wasserstein Energy-Based Sliced Wasserstein
Closest papers by meaning
- Automatic Controllable Colorization via Imagination (CVPR 2024 · similarity 0.86)
- Color Alignment in Diffusion (CVPR 2025 · similarity 0.86)
- Free-Lunch Color-Texture Disentanglement for Stylized Image Generation (NeurIPS 2025 · similarity 0.85)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.84)
- MangaNinja: Line Art Colorization with Precise Reference Following (CVPR 2025 · similarity 0.84)
- Image Referenced Sketch Colorization Based on Animation Creation Workflow (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.