Research map / Video / Video generation
AniDoc: Animation Creation Made Easier
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds AniDoc, a video-diffusion (SVD-based) line art colorization model that colors sketch sequences from a single character design reference using correspondence point maps, binarized sketches with background augmentation, and sparse-sketch interpolation training.
Problem
Automating reference-based colorization of anime line art videos is hard because character design references are misaligned with sketch poses/angles, binarized sketches lose color cues (non-binarized training leaks color), and previous methods need dense sketches and colored first frames.
Approach
Explicit keypoint correspondence point maps (LightGlue+SIFT in training, X-Pose+DIFT at inference) injected via a control branch into SVD, combined with binarized training sketches, background augmentation, and a two-stage sparse-sketch training that learns interpolation from start/end sketches and tracked trajectories.
Key results
- AniDoc reaches PSNR 19.23, SSIM 0.7720, LPIPS 0.1704, FID 54.33, FVD 230.18, beating the best baseline LVCD+IP (16.52 PSNR, FID 118.28, FVD 496.45).
- Removing binarization and background augmentation raises FVD to 503.23 and FID to 94.89 versus 230.18 and 54.33 for the full model.
- Removing correspondence matching gives PSNR 17.80 and FVD 273.86 versus 19.23 and 230.18 for the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AniDoc instead of the strongest baseline, LVCD + IP | Reference-based line art video colorization · AniDoc test set | PSNR: 16.52 -> 19.23 | Helped |
| use AniDoc instead of the strongest baseline, LVCD | Reference-based line art video colorization · AniDoc test set | SSIM: 0.6446 -> 0.7720 | Helped |
| use AniDoc instead of the strongest baseline, LVCD | Reference-based line art video colorization · AniDoc test set | LPIPS: 0.2580 -> 0.1704 | Helped |
| use AniDoc instead of the strongest baseline, ToonCrafter + IP | Reference-based line art video colorization · AniDoc test set | FID: 110.48 -> 54.33 | Helped |
| use AniDoc instead of the strongest baseline, ToonCrafter + IP | Reference-based line art video colorization · AniDoc test set | FVD: 492.10 -> 230.18 | Helped |
| add binarization and background augmentation (ablation) | AniDoc · AniDoc test set | PSNR: 15.41 -> 19.23 | Helped |
| add binarization and background augmentation (ablation) | AniDoc · AniDoc test set | SSIM: 0.6639 -> 0.7720 | Helped |
| add binarization and background augmentation (ablation) | AniDoc · AniDoc test set | LPIPS: 0.2471 -> 0.1704 | Helped |
| add binarization and background augmentation (ablation) | AniDoc · AniDoc test set | FID: 94.89 -> 54.33 | Helped |
| add binarization and background augmentation (ablation) | AniDoc · AniDoc test set | FVD: 503.23 -> 230.18 | Helped |
| add correspondence matching (ablation) | AniDoc · AniDoc test set | PSNR: 17.80 -> 19.23 | Helped |
| add correspondence matching (ablation) | AniDoc · AniDoc test set | SSIM: 0.7068 -> 0.7720 | Helped |
Surprising findings
- Non-binarized sketches leak color: LVCD given an empty reference still produced colors resembling ground truth.
Limitations
- Limited to 256x256 training resolution in the first stage due to GPU memory constraints.
- Evaluation removes backgrounds and resizes to 256x256, and the reference design lacks background information.
- Sparse-sketch interpolation relies on keypoint trajectories and is suited to simple motions.
Experimental setup
Datasets: Sakuga-42M
Benchmarks: 200-clip anime test set from 10 eras/styles
Metrics: PSNR SSIM LPIPS FID FVD
Baselines: LVCD LVCD+IP-Adapter ToonCrafter+IP-Adapter ID-animator
Builds on: Stable Video Diffusion ControlNet LightGlue DIFT CoTracker
Closest papers by meaning
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.88)
- Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters (CVPR 2025 · similarity 0.86)
- Image Referenced Sketch Colorization Based on Animation Creation Workflow (CVPR 2025 · similarity 0.86)
- Learning Inclusion Matching for Animation Paint Bucket Colorization (CVPR 2024 · similarity 0.86)
- MangaNinja: Line Art Colorization with Precise Reference Following (CVPR 2025 · similarity 0.86)
- Breathing Life Into Sketches Using Text-to-Video Priors (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.