Research map / Image generation / Editing and style transfer
Image Referenced Sketch Colorization Based on Animation Creation Workflow
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a diffusion-based image-referenced sketch colorization framework with split cross-attention using foreground/background LoRAs and switchable inference modes to eliminate spatial entanglement artifacts.
Problem
Image-referenced sketch colorization for animation production suffers from spatial entanglement and artifacts when the reference image's composition mismatches the sketch.
Approach
Separates foreground and background of the reference image via spatial masks and processes each with separate trainable LoRA key/value weights in cross-attention within a single forward pass, plus a recovery transformer for background embeddings and switchable LoRA inference modes (Vanilla, Bg2Fig, Fig2Fig).
Key results
- Achieves 50K-FID of 6.83 versus 9.53 for ColorizeDiffusion and 38.9 for IP-Adapter.
- Obtains MS-SSIM 0.6002 and CLIP similarity 0.8829, best among compared methods; PSNR 28.91 is second to Yan et al. (29.25).
- User study with 40 participants shows the method is preferred over all six baselines in overall quality and geometric preservation (Kruskal-Wallis p<0.01).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add split cross-attention with trainable foreground/background LoRAs and recovery transformer | sketch colorization diffusion framework · Validation 50K FID | FID: 9.5276 -> 6.8327 | Helped |
| add split cross-attention with LoRAs and recovery transformer (vs ColorizeDiffusion base) | sketch colorization framework · 5K TPS-deformed pairs | MS-SSIM: 0.5913 -> 0.6002 | Helped |
| proposed framework vs IP-Adapter-style adapters | sketch colorization · Validation 50K FID | FID: 38.9184 -> 6.8327 | Helped |
| GAN-based method achieves higher PSNR than proposed method; proposed ranks second | sketch colorization · 5K TPS-deformed pairs | PSNR: 29.2483 vs 28.9144 | Hurt |
| split cross-attention without trainable LoRAs and recovery transformer (ablation) | baseline framework · Qualitative ablation (Figure 6) | spatial artifacts | Helped |
| add trainable LoRAs to split cross-attention (ablation) | split cross-attention framework · Qualitative ablation (Figure 6) | color saturation and details | Helped |
| add recovery transformer on background embeddings (ablation) | split cross-attention with trainable LoRAs · Qualitative ablation (Figure 6) | artifacts | Helped |
| Fig2Fig inference mode instead of Vanilla/Bg2Fig (ablation) | inference · Qualitative (Figure 7) | spatial entanglement elimination | Mixed |
| add spatial masks plus separated trainable LoRAs (user study comparison vs six baselines) | sketch colorization · User study (40 participants) | user preference | Helped |
Surprising findings
- The GAN-based Yan et al. method achieves the best PSNR despite much poorer visual quality, which the authors attribute to PSNR favoring average-like outputs.
Limitations
- Performance strongly depends on the quality of the extracted foreground masks.
- Fig2Fig mode is less suitable for inpainting character sketches where regions without mask guidance lack reference information.
Experimental setup
Datasets: Danbooru2021
Benchmarks: Danbooru2021 validation set (50K pairs) 5K TPS-deformed test triples
Metrics: FID PSNR MS-SSIM CLIP similarity user preference
Baselines: ColorizeDiffusion IP-Adapter InstantStyle T2I-Adapter AnimeDiffusion Yan et al. GAN-based colorization
Builds on: Stable Diffusion ColorizeDiffusion IP-Adapter ControlNet DPM++
Closest papers by meaning
- MangaNinja: Line Art Colorization with Precise Reference Following (CVPR 2025 · similarity 0.89)
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.87)
- Learning Inclusion Matching for Animation Paint Bucket Colorization (CVPR 2024 · similarity 0.87)
- Automatic Controllable Colorization via Imagination (CVPR 2024 · similarity 0.86)
- AniDoc: Animation Creation Made Easier (CVPR 2025 · similarity 0.86)
- SSIMBaD: Sigma Scaling with SSIM-Guided Balanced Diffusion for AnimeFace Colorization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.