Research map / Image generation / Editing and style transfer
MangaNinja: Line Art Colorization with Precise Reference Following
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MangaNinja, a diffusion-based reference-guided line art colorization model with patch shuffling for local correspondence learning and PointNet-based point control, trained on anime video frames and evaluated on a self-collected 200-pair benchmark.
Problem
Reference-based line art colorization often fails on large pose/structure discrepancies between reference and line art and lacks precise control, losing fine character details.
Approach
A dual-branch Stable Diffusion 1.5 design (Reference U-Net and Denoising U-Net with shared key/value concatenation) trained with progressive patch shuffling to force local matching, plus a PointNet-encoded point-pair control scheme with multiple classifier-free guidance terms for fine-grained interactive color matching.
Key results
- Full MangaNinja with point guidance reaches DINO 69.91, CLIP 90.02, PSNR 21.34, MS-SSIM 0.972, outperforming BasicPBC (PSNR 17.58) and AnyDoor* (PSNR 16.24) on the self-collected benchmark.
- Without any point guidance, MangaNinja already achieves PSNR 20.37 and CLIP 88.34, above all baselines.
- Ablation shows progressive patch shuffle gives the largest gain for automatic matching (PSNR 18.02 to 18.61 alone, and multi-CFG 19.72), with all components combined giving best results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add progressive patch shuffle (ablation) | MangaNinja without point guidance · MangaNinja benchmark | PSNR: 18.02 -> 19.72 | Helped |
| add condition dropping (ablation) | MangaNinja without point guidance · MangaNinja benchmark | PSNR: 18.02 -> 18.61 | Helped |
| add point control (ablation) | MangaNinja · MangaNinja benchmark | PSNR: 20.37 -> 21.34 | Helped |
| use the full training strategy set, including point control, condition dropping, patch shuffle, multi-CFG, and two-stage training (ablation) | MangaNinja · MangaNinja benchmark | PSNR: 18.02 -> 21.34 | Helped |
| use MangaNinja instead of the strongest baseline for DINO similarity | MangaNinja benchmark comparison · MangaNinja benchmark | DINO similarity: 63.79 -> 69.91 | Helped |
| use MangaNinja instead of the strongest baseline for CLIP similarity | MangaNinja benchmark comparison · MangaNinja benchmark | CLIP similarity: 83.91 -> 90.02 | Helped |
| use MangaNinja instead of the strongest baseline for PSNR | MangaNinja benchmark comparison · MangaNinja benchmark | PSNR: 17.58 -> 21.34 | Helped |
| use MangaNinja instead of the strongest baseline for MS-SSIM | MangaNinja benchmark comparison · MangaNinja benchmark | MS-SSIM: 0.894 -> 0.972 | Helped |
| use MangaNinja instead of the strongest baseline for LPIPS | MangaNinja benchmark comparison · MangaNinja benchmark | LPIPS: 0.27 -> 0.21 | Helped |
Surprising findings
- BasicPBC, a non-generative method, outperforms generative baselines on pixel-level PSNR and MS-SSIM though it is weaker on feature similarity metrics.
Limitations
- Evaluation relies on a self-collected 200-pair benchmark, limiting comparability with other datasets.
- Point control requires user-specified correspondences and the model depends on automatic matching ability learned from video data.
Experimental setup
Datasets: sakuga-42m
Benchmarks: self-collected 200-pair line art colorization benchmark
Metrics: DINO similarity CLIP similarity PSNR MS-SSIM LPIPS MSE at matching points
Baselines: BasicPBC IP-Adapter AnyDoor ToonCrafter
Builds on: Stable Diffusion 1.5 ReferenceNet LightGlue LineartAnimeDetector
Closest papers by meaning
- Image Referenced Sketch Colorization Based on Animation Creation Workflow (CVPR 2025 · similarity 0.89)
- SSIMBaD: Sigma Scaling with SSIM-Guided Balanced Diffusion for AnimeFace Colorization (NeurIPS 2025 · similarity 0.87)
- Learning Inclusion Matching for Animation Paint Bucket Colorization (CVPR 2024 · similarity 0.87)
- Automatic Controllable Colorization via Imagination (CVPR 2024 · similarity 0.87)
- AniDoc: Animation Creation Made Easier (CVPR 2025 · similarity 0.86)
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.