Research map / Video / Video generation
ControlVideo: Training-free Controllable Text-to-video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ControlVideo, a training-free controllable text-to-video method that inflates ControlNet with fully cross-frame attention, an interleaved-frame interpolation smoother, and a hierarchical key-frame sampler for long videos.
Problem
Text-to-video diffusion models are costly to train, and training-free controllable video generation from motion sequences suffers from appearance inconsistency, artifacts under large motion, and structural flicker, especially for long videos.
Approach
Concatenate all frames into a single "large image" for self-attention (fully cross-frame interaction) so the pretrained ControlNet/Stable Diffusion behaviour is preserved, smooth structural flicker by RIFE-based interpolation of alternating frames at a few denoising timesteps, and generate long videos by pre-generating key frames then synthesizing clips between them.
Key results
- On 125 motion-prompt pairs with depth maps, ControlVideo reaches 97.22% frame consistency and 31.81% prompt consistency, versus 95.99% and 31.69% for Text2Video-Zero.
- Interleaved-frame smoother raises frame consistency from 95.36% (fully cross-frame) to 96.83% at a cost of about 0.5 min extra.
- Generates short videos in about 2 minutes and 100-frame long videos in about 10 minutes on one NVIDIA RTX 2080Ti.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ControlVideo instead of Text2Video-Zero with Canny Edge conditioning | ControlNet-based video generation · 125 motion-prompt pairs, Canny Edge | Frame Consistency (%): 95.17 -> 96.83 | Helped |
| use ControlVideo instead of Text2Video-Zero with Canny Edge conditioning | ControlNet-based video generation · 125 motion-prompt pairs, Canny Edge | Prompt Consistency (%): 30.74 -> 30.75 | Helped |
| use ControlVideo instead of Text2Video-Zero with Depth Map conditioning | ControlNet-based video generation · 125 motion-prompt pairs, Depth Map | Frame Consistency (%): 95.99 -> 97.22 | Helped |
| use ControlVideo instead of Text2Video-Zero with Depth Map conditioning | ControlNet-based video generation · 125 motion-prompt pairs, Depth Map | Prompt Consistency (%): 31.69 -> 31.81 | Helped |
| replace no frame interaction with fully cross-frame interaction (ablation) | ControlNet video generation · Canny Edge, cross-frame mechanism ablation | Frame Consistency (%): 89.94 -> 95.36 | Helped |
| replace first-only attention with fully cross-frame interaction (ablation) | ControlNet video generation · Canny Edge, cross-frame mechanism ablation | Frame Consistency (%): 94.92 -> 95.36 | Helped |
| replace sparse-causal attention with fully cross-frame interaction (ablation) | ControlNet video generation · Canny Edge, cross-frame mechanism ablation | Frame Consistency (%): 95.06 -> 95.36 | Helped |
| add interleaved-frame smoother to fully cross-frame interaction (ablation) | ControlVideo · Canny Edge, smoother ablation | Frame Consistency (%): 95.36 -> 96.83 | Helped |
| add interleaved-frame smoother to fully cross-frame interaction (ablation) | ControlVideo · Canny Edge, smoother ablation | Prompt Consistency (%): 30.76 -> 30.79 | No clear effect |
Surprising findings
- Sparser cross-frame attention variants (first-only, sparse-causal) increase the query-key discrepancy relative to ControlNet, which degrades quality, so denser interaction worked better.
Limitations
- Struggles to produce videos that deviate from the input motion sequence; cannot easily follow text prompts that require different motion.
- Fully cross-frame attention adds 1-2x time cost and memory, limited to short clips (under 16 frames) without the hierarchical sampler.
Experimental setup
Datasets: DAVIS
Benchmarks: DAVIS-based 125 motion-prompt pairs
Metrics: frame consistency (CLIP) prompt consistency (CLIP) user preference time cost
Baselines: Tune-A-Video Text2Video-Zero Follow-Your-Pose
Builds on: ControlNet Stable Diffusion DDIM RIFE Tune-A-Video
Closest papers by meaning
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.90)
- A Recipe for Scaling up Text-to-Video Generation with Text-free Videos (CVPR 2024 · similarity 0.89)
- ConditionVideo: Training-Free Condition-Guided Video Generation (AAAI 2024 · similarity 0.89)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
- ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (CVPR 2025 · similarity 0.89)
- CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.