Research map / Video / Video generation
FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes ModelScope text-to-video diffusion with a LoRA on synthetic sketch animations and adds DDIM-inversion reference-frame noise refinement and dual-attention composition to animate raster sketches from a drawing plus text prompt.
Problem
Generating fluid, unconstrained sketch animations from a single static drawing and text prompt, avoiding stroke-constrained vector animation and per-sample optimization costs while preserving sketch identity.
Approach
Treat sketch animation as raster video generation: a LoRA-adapted T2V model whose first frame is the DDIM-inverted input sketch, with iterative alignment of sampled noise to the reference and spatial/temporal attention composition with reference query-key pairs to preserve identity while allowing motion.
Key results
- Ours achieves T2V alignment 0.172 vs Live-Sketch 0.142 and DynamiCrafter 0.127 (X-CLIP), while Live-Sketch has higher S2V consistency (0.965 vs 0.956).
- User study MOS of 0.70 for Ours vs 0.63 for Live-Sketch and 0.53 for T2V LoRA.
- Removing spatial and temporal attention composition drops S2V consistency from 0.956 to 0.876.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlipSketch instead of Live-Sketch | Sketch animation · Sketch animation | T2V Alignment: 0.142 ± 0.005 -> 0.172 ± 0.002 | Helped |
| Use FlipSketch instead of Live-Sketch | Sketch animation · Sketch animation | S2V Consistency: 0.965 ± 0.003 -> 0.956 ± 0.004 | Hurt |
| Use FlipSketch instead of T2V LoRA | Sketch animation · Sketch animation | Faithfulness: 0.27 -> 0.54 | Helped |
| Use FlipSketch instead of T2V LoRA | Sketch animation · Sketch animation | MOS: 0.53 -> 0.70 | Helped |
| Set λ to 0 to increase motion (ablation) | FlipSketch · Sketch animation | T2V Alignment: 0.172 ± 0.002 -> 0.174 ± 0.001 | Helped |
| Set λ to 0 to increase motion (ablation) | FlipSketch · Sketch animation | S2V Consistency: 0.956 ± 0.004 -> 0.949 ± 0.002 | Hurt |
| Set λ to 1 (ablation) | FlipSketch · Sketch animation | S2V Consistency: 0.956 ± 0.004 -> 0.968 ± 0.003 | Helped |
| Set λ to 1 (ablation) | FlipSketch · Sketch animation | T2V Alignment: 0.172 ± 0.002 -> 0.170 ± 0.001 | Hurt |
| Remove temporal and spatial attention composition (ablation) | FlipSketch · Sketch animation | S2V Consistency: 0.956 ± 0.004 -> 0.876 ± 0.004 | Hurt |
| Remove temporal and spatial attention composition (ablation) | FlipSketch · Sketch animation | Consistency: 0.54 -> 0.20 | Hurt |
| Remove iterative frame alignment (ablation) | FlipSketch · Sketch animation | S2V Consistency: 0.956 ± 0.004 -> 0.952 ± 0.004 | No clear effect |
Surprising findings
- Live-Sketch achieves higher sketch-to-video consistency than the proposed method on CLIP metrics, despite lower motion range.
Limitations
- Stylistic resemblance to CLIPasso-style sketches due to uni-modal training data
- Poor handling of abstract or geometrically irregular sketches
- Inherits motion artifacts such as extra limbs from ModelScope pre-training
Experimental setup
Datasets: synthetic vector sketch animations generated via Live-Sketch
Benchmarks: in-house sketch animation prompts with CLIP-based metrics
Metrics: S2V consistency (CLIP) T2V alignment (X-CLIP) user study consistency faithfulness MOS
Baselines: Live-Sketch SVD DynamiCrafter T2V LoRA Dynamic Typography
Builds on: ModelScope T2V Live-Sketch TF-ICON null-text inversion LoRA
Closest papers by meaning
- Breathing Life Into Sketches Using Text-to-Video Priors (CVPR 2024 · similarity 0.89)
- SketchVideo: Sketch-based Video Generation and Editing (CVPR 2025 · similarity 0.87)
- AirSketch: Generative Motion to Sketch (NeurIPS 2024 · similarity 0.87)
- SketchAgent: Language-Driven Sequential Sketch Generation (CVPR 2025 · similarity 0.86)
- StickMotion: Generating 3D Human Motions by Drawing a Stickman (CVPR 2025 · similarity 0.86)
- One-Shot Reference-based Structure-Aware Image to Sketch Synthesis (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.