Research map / Video / Video generation

SketchVideo: Sketch-based Video Generation and Editing

CVPR 2025 · Video › Video generation · approach: Diffusion-based human image animation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SketchVideo, a DiT-based (CogVideoX-2b) sketch-conditioned framework for video generation from one or two keyframe sketches and for sketch-driven local editing of real or synthetic videos.

Problem

Text prompts cannot precisely control global layout and geometry in video generation, and existing methods either need dense per-frame conditions or are memory-heavy; local sketch-based editing with motion preservation and unedited-region fidelity is also hard.

Approach

A memory-efficient skip residual control network with only 5 trainable DiT-block copies (predicting residuals for blocks 0,6,12,18,24) processes only sketches (no white placeholders), an inter-frame attention propagates sparse keyframe sketch features across frames, and a video insertion module plus inference-time latent fusion enables sketch-based editing.

Key results

• Sketch-based generation: LPIPS 27.56 and CLIP 98.31 vs. SparseCtrl's 44.85 / 96.48 and AMT's 29.17 / 96.12 (Table 1).
• Sketch-based editing: PSNR 36.48 and LPIPS 9.74 vs. AnyV2V's 13.68 / 11.92 and InsV2V's 16.84 / 13.61 (Table 2).
• User study with 20 participants ranked SketchVideo best on fidelity, consistency, and realism for both generation and editing.

Measured effects reported in the paper

Surprising findings

• Removing image data from training causes significant geometry mismatch with input sketches, showing image data helps video sketch control despite limited video data.
• CLIP scores barely differ across ablations, with the authors noting human eyes cannot recognize the temporal coherence difference.

Limitations

• Limited by the capability of the pretrained text-to-video model.
• Struggles with complex scenarios such as human hands and multi-object interactions.
• Generates short clips rather than long videos; geometry-only control without appearance (color) control.

Experimental setup

Datasets: OpenVid LAION WebVid-10M (SparseCtrl pretrained only)

Benchmarks: OpenVid (200 random examples for generation) 10 hand-drawn editing examples

Metrics: LPIPS CLIP similarity PSNR user study ranking

Baselines: AMT SparseCtrl Ctrl-CogVideo ControlNet InsV2V AnyV2V

Builds on: CogVideoX ControlNet PIXART-δ SparseCtrl DDIM inversion

Closest papers by meaning

• FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations (CVPR 2025 · similarity 0.87)
• VIRES: Video Instance Repainting via Sketch and Text Guided Generation (CVPR 2025 · similarity 0.87)
• Breathing Life Into Sketches Using Text-to-Video Priors (CVPR 2024 · similarity 0.86)
• ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.86)
• Keyframe-Guided Creative Video Inpainting (CVPR 2025 · similarity 0.86)
• Generative Inbetweening through Frame-wise Conditions-Driven Video Generation (CVPR 2025 · similarity 0.86)