Research map / Image generation / Fast diffusion sampling
Color Alignment in Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes color alignment inside diffusion sampling/training that projects noisy samples or predicted x0 onto a given color pattern, enabling fine-grained color-conditioned image synthesis in re-trained, fine-tuned, and zero-shot settings.
Problem
Fine-grained control of generated pixel colors (accuracy, completeness, disentanglement from structure) in diffusion models, where existing conditioning via text or reference images lets outputs drift outside the desired color distribution.
Approach
Map diffused samples (or predicted x0 in a zero-shot variant) onto the nearest colors of a color pattern at each diffusion step, after randomly permuting the pattern's pixel locations to remove spatial information, so the denoiser operates within a color manifold while keeping structure free.
Key results
- On Oxford-flower image diffusion, the re-trained color-aligned model reaches FID 57.5 vs 63.9 for DDPM, with CD-A 0.00 vs 40.6.
- In latent diffusion with manual color conditions, the fine-tuned version attains CD-A 73.9 and CLIPScore 29.0 vs Stable Diffusion's 166 and 27.3 (note: CD-A is worse than baseline for this setting, so comparison is mixed).
- Zero-shot variant on Oxford-flower gives FID 50.0 and CD-A 5.59 versus DDPM's 63.9 and 40.6, at about 6.76x inference cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add color mapping to the diffusion process (ablation) | Image diffusion with color-condition input · Oxford-flower and Microsoft-emoji | FID: 57.9; 88.1 -> 57.5; 45.6 | Mixed |
| add color-condition input to the diffusion model (ablation) | Image diffusion with color mapping · Oxford-flower and Microsoft-emoji | FID: 59.8; 45.7 -> 57.5; 45.6 | Mixed |
| add color mapping and color-condition input (ablation) | Image diffusion · Oxford-flower and Microsoft-emoji | FID: 63.9; 149 -> 57.5; 45.6 | Helped |
| add color mapping and color-condition input (ablation) | Image diffusion · Oxford-flower and Microsoft-emoji | CD-A: 40.6; 42.4 -> 0.00; 0.00 | Helped |
| add color mapping and color-condition input (ablation) | Image diffusion · Oxford-flower and Microsoft-emoji | CD-C: 42.6; 39.0 -> 4.12; 3.65 | Helped |
| use latent fine-tuning instead of the pretrained model | Stable Diffusion · Manually sampled color condition | FID: 72.6 -> 86.7 | Hurt |
| use latent fine-tuning instead of the pretrained model | Stable Diffusion · In-the-wild image color condition | FID: 72.6 -> 69.4 | Helped |
Surprising findings
- Applying the zero-shot color mapping to the training pipeline (g instead of f) corrupts the noise distribution and was found not applicable.
- Color-aligned models sometimes achieve better FID than the unconstrained baseline despite the color constraint.
Limitations
- Spatial information in color conditions is discarded, so spatial layout of colors cannot be controlled.
- Zero-shot variant yields less fine-grained detail and blurry lighting, and its inference cost is several times higher.
- Evaluation relies on FID, CLIPScore and Chamfer distance; no human study is reported.
Experimental setup
Datasets: Oxford-flower Microsoft-emoji Text-to-image-2M
Benchmarks: Oxford-flower Microsoft-emoji Text-to-image-2M subset
Metrics: FID CLIPScore Chamfer Distance accuracy (CD-A) Chamfer Distance completeness (CD-C) inference cost
Baselines: DDPM Stable Diffusion IP-Adapter T2I-Adapter Style-Aligned ControlNet-Color
Builds on: DDPM Stable Diffusion classifier-free guidance
Closest papers by meaning
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.88)
- Angle Domain Guidance: Latent Diffusion Requires Rotation Rather Than Extrapolation (ICML 2025 · similarity 0.87)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.87)
- FAM Diffusion: Frequency and Attention Modulation for High-Resolution Image Generation with Stable Diffusion (CVPR 2025 · similarity 0.87)
- Stitched Value Model for Diffusion Alignment (NeurIPS 2026 · similarity 0.87)
- DiffuseHigh: Training-Free Progressive High-Resolution Image Synthesis Through Structure Guidance (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.