Research map / Video / Video generation
Breathing Life Into Sketches Using Text-to-Video Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Animates a single-subject vector sketch from a text prompt by distilling the motion prior of ModelScope text-to-video with SDS through a neural displacement field split into local and global affine branches.
Problem
Animating sketches is laborious and requires professional skill; existing tools need skeletal annotations or reference motions, and pixel-based image-to-video models fail to preserve sketch appearance.
Approach
Represent the sketch as cubic Bezier strokes and optimize a neural displacement field with SDS against a pretrained text-to-video model, decomposing motion into an unconstrained local offset branch and a per-frame global affine transformation branch to keep motion smooth while preserving shape.
Key results
- Ours achieves 0.965 sketch-to-video consistency vs 0.876 for VideoCrafter and 0.779 for ModelScope.
- Ours achieves 0.142 text-to-video alignment (X-CLIP) vs 0.124 for VideoCrafter.
- Ablation: removing the neural network (0.926) or global path (0.936) lowers sketch consistency relative to full model (0.965); user study with 31 participants over 30 pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use vector-based sketch animation guided by a text-to-video prior instead of VideoCrafter image-to-video | Ours · Sketch-to-video evaluation | sketch-to-video consistency: 0.876 ± 0.007 -> 0.965 ± 0.003 | Helped |
| use vector-based sketch animation guided by a text-to-video prior instead of VideoCrafter image-to-video | Ours · Sketch-to-video evaluation | text-to-video alignment: 0.124 ± 0.005 -> 0.142 ± 0.005 | Helped |
| add a neural network rather than directly optimize control-point coordinates (ablation) | Ours · Ablation evaluation | sketch-to-video consistency: 0.926 ± 0.007 -> 0.965 ± 0.003 | Helped |
| add a neural network rather than directly optimize control-point coordinates (ablation) | Ours · Ablation evaluation | text-to-video alignment: 0.142 ± 0.005 -> 0.142 ± 0.005 | No clear effect |
| add the global-motion path (ablation) | Ours · Ablation evaluation | sketch-to-video consistency: 0.936 ± 0.006 -> 0.965 ± 0.003 | Helped |
| add the global-motion path (ablation) | Ours · Ablation evaluation | text-to-video alignment: 0.140 ± 0.005 -> 0.142 ± 0.005 | No clear effect |
| add the local-motion path (ablation) | Ours · Ablation evaluation | sketch-to-video consistency: 0.970 ± 0.002 -> 0.965 ± 0.003 | Hurt |
| add the local-motion path (ablation) | Ours · Ablation evaluation | text-to-video alignment: 0.140 ± 0.004 -> 0.142 ± 0.005 | No clear effect |
Surprising findings
- Text-to-video models that struggle to generate sketch videos directly still serve as a useful motion prior for animating vector sketches.
- ModelScope scores poorly on sketch consistency when generating pixel sketches, while the vector representation yields much higher consistency.
Limitations
- Struggles with multi-object sketches or scene sketches.
- Trade-off between motion quality and sketch fidelity; shape changes for unusual sketches.
- Limited to motions the text-to-video prior can generate and inherits its biases and artifacts.
Experimental setup
Datasets: CLIPasso-generated sketches TU-Berlin sketch dataset (evaluation)
Benchmarks: CLIP sketch-to-video consistency on 30 generated animations X-CLIP text-to-video alignment
Metrics: CLIP cosine similarity (sketch-to-video consistency) X-CLIP similarity (text-to-video alignment) user preference (two-alternative forced choice)
Baselines: ZeroScope ModelScope image-to-video VideoCrafter Gen-2 Animated Drawings
Builds on: score distillation sampling (DreamFusion) CLIPasso DiffVG ModelScope text-to-video
Closest papers by meaning
- FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations (CVPR 2025 · similarity 0.89)
- AirSketch: Generative Motion to Sketch (NeurIPS 2024 · similarity 0.87)
- SketchVideo: Sketch-based Video Generation and Editing (CVPR 2025 · similarity 0.86)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.86)
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.85)
- Recovering Dynamic 3D Sketches from Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.