Research map / Image generation / Editing and style transfer
Automatic Controllable Colorization via Imagination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an automatic colorization framework that uses ControlNet-conditioned Stable Diffusion to imagine multiple colorful references, composes an optimal reference via Semantic-SAM segments and DINOv2 nearest-neighbor matching, and colorizes with a UniColor-based module, enabling iterative user edits.
Problem
Automatic colorization of grayscale images often yields grayish, desaturated, or inconsistent results and offers no way for users to iteratively edit or control the output, unlike human experts or exemplar-based methods that need a manually chosen reference.
Approach
An imagination module synthesizes multiple semantically similar, structurally aligned reference candidates from a pretrained diffusion model (ControlNet with Canny/HED conditions and BLIP-2 captions); a reference refinement module composes a per-segment optimal reference via DINOv2 feature nearest neighbors over Semantic-SAM segments, and users can swap segments to edit the result.
Key results
- Achieves the highest colorfulness (CF) on all three test sets: 47.2 on COCO-Stuff, 48.8 on ImageNet, and 48.9 on in-the-wild, versus 46.2 for the next best on COCO-Stuff.
- In a 30-user study, the method was preferred at 43.3% against DDColor, with the other baselines receiving lower shares, the highest being 26.7% for DISCO.
- Reference refinement composition lowers FID from 8.28 to 7.21 on COCO-Stuff and raises CF from 43.87 to 47.18 relative to a single reference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine Canny and HED conditions rather than use either condition alone (ablation) | Imagination Module · COCO-stuff validation | CF: 46.99, 45.28 -> 47.18 | Helped |
| use depth conditioning instead of combined Canny and HED conditions (ablation) | Imagination Module · COCO-stuff validation | CF: 47.18 -> 42.81 | Hurt |
| use semantic conditioning instead of combined Canny and HED conditions (ablation) | Imagination Module · COCO-stuff validation | CF: 47.18 -> 45.02 | Hurt |
| increase reference candidates from N=2 to N=8 (ablation) | Imagination Module · COCO-stuff validation | CF: 38.81 -> 47.18 | Helped |
| increase reference candidates from N=8 to N=16 (ablation) | Imagination Module · COCO-stuff validation | CF: 47.18 -> 47.82 | Helped |
| increase reference candidates from N=8 to N=32 (ablation) | Imagination Module · COCO-stuff validation | CF: 47.18 -> 48.69 | Helped |
| increase segments from NS=10 to NS=40 (ablation) | Reference Refinement Module · COCO-stuff validation | CF: 47.18 -> 42.09 | Hurt |
| increase hint colors from NH=10 to NH=50 (ablation) | Colorization Module · COCO-stuff validation | CF: 47.18 -> 45.53 | Hurt |
| compose a refined reference instead of using a single reference (ablation) | Reference Refinement Module · COCO-stuff validation | CF: 43.87 -> 47.18 | Helped |
| optimize coarse hint colors into fine hint colors by clustering (ablation) | Colorization Module · COCO-stuff validation | color bleeding | Helped |
| use Ours instead of DDColor | automatic colorization methods · COCO-stuff validation | CF: 44.7 -> 47.2 | Helped |
| use Ours instead of DDColor | automatic colorization methods · ImageNet testing split ctest | CF: 42.2 -> 48.8 | Helped |
Surprising findings
- Directly applying ControlNet to colorization produces inconsistency, grayish results, and color bleeding, which motivates the composition strategy rather than using generated images directly.
Limitations
- Struggles with images containing many identical instances, such as numerous humans, producing artifacts and inconsistencies.
- Inference cost is high; about 8 seconds per 512x512 image even with LCM-LoRA acceleration.
Experimental setup
Datasets: COCO-stuff ImageNet
Benchmarks: COCO-stuff validation ImageNet test In-the-wild internet photos
Metrics: FID Colorfulness PSNR SSIM LPIPS user preference
Baselines: Deoldify Su et al. Coltran GCP CT2 ColorFormer BigColor UniColor DISCO DDColor
Builds on: ControlNet UniColor Stable Diffusion Semantic-SAM DINOv2
Closest papers by meaning
- Paint Anything: Toward Any-Color Controllable Image Generation and Editing (NeurIPS 2026 · similarity 0.88)
- Colorization Transformer (ICLR 2021 · similarity 0.88)
- MangaNinja: Line Art Colorization with Precise Reference Following (CVPR 2025 · similarity 0.87)
- Image Referenced Sketch Colorization Based on Animation Creation Workflow (CVPR 2025 · similarity 0.86)
- Color Conditional Generation with Sliced Wasserstein Guidance (NeurIPS 2025 · similarity 0.86)
- Enabling Region-Specific Control via Lassos in Point-Based Colorization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.