Research map / Video / Video generation
ConditionVideo: Training-Free Condition-Guided Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ConditionVideo, a training-free condition-guided text-to-video method built on ControlNet and Stable Diffusion 1.5, using disentangled background/foreground noise, sparse bi-directional spatial-temporal attention, and a 3D control branch.
Problem
Text-to-video generation with control conditions (pose, canny, depth, segmentation) and realistic dynamic backgrounds usually requires costly training or per-video tuning; existing training-free methods produce poor background dynamics or weak condition alignment.
Approach
Disentangles motion into condition-guided foreground (control branch with fixed per-video noise) and scenery background (UNet branch from DDIM-inverted reference video or fixed noise), and replaces self-attention with sparse bi-directional spatial-temporal attention that attends to frames with gap 3, with the control branch inflated to 3D so it also uses neighboring frames.
Key results
- Pose condition: 99.02% frame consistency, 33.03 CLIP score, 83.12% pose accuracy, versus ControlNet 94.22 / 32.97 / 79.51 and Text2Video-Zero 98.82 / 32.84 / 78.50.
- Tune-A-Video reaches only 26.13% pose accuracy despite requiring per-video fine-tuning.
- sBiST-Attn reaches 99.02% frame consistency in 1m30s versus 3m37s for full attention (99.03%), and 43s for S-C Attn (98.77%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ConditionVideo instead of the strongest listed baseline, Text2Video-Zero | video generation under pose conditioning · Pose | FC (%): 98.82 -> 99.02 | Helped |
| use ConditionVideo instead of the strongest listed baseline, ControlNet | video generation under pose conditioning · Pose | CS: 32.97 -> 33.03 | Helped |
| use ConditionVideo instead of the strongest listed baseline, ControlNet | video generation under pose conditioning · Pose | PA (%): 79.51 -> 83.12 | Helped |
| use ConditionVideo instead of Text2Video-Zero | video generation with canny conditioning · Canny | FC (%): 97.44 -> 97.64 | Helped |
| use ConditionVideo instead of ControlNet | video generation with canny conditioning · Canny | CS: 29.65 -> 29.76 | Helped |
| use ConditionVideo instead of Text2Video-Zero | video generation with depth conditioning · Depth | FC (%): 97.46 -> 97.65 | Helped |
| use ConditionVideo instead of ControlNet | video generation with depth conditioning · Depth | CS: 30.16 -> 30.54 | Helped |
| add sBiST-Attn instead of non-temporal self-attention (ablation) | ConditionVideo · Image-level noise; pose condition | FC (%): 94.22 -> 99.02 | Helped |
| add bidirectional frame attention rather than sparse-causal attention (ablation) | ConditionVideo · Image-level noise; pose condition | FC (%): 98.77 -> 99.02 | Helped |
| replace sBiST-Attn with dense full attention (ablation) | ConditionVideo · Image-level noise; pose condition | FC (%): 99.02 -> 99.03 | No clear effect |
| replace the 3D control branch with frame-wise 2D control (ablation) | ConditionVideo · Pose | PA (%): 81.26 -> 83.12 | Hurt |
| replace the 3D control branch with frame-wise 2D control (ablation) | ConditionVideo · Pose | FC (%): 99.03 -> 99.02 | No clear effect |
Surprising findings
- Dense full attention gives almost no frame-consistency gain over sBiST-Attn despite more than double the generation time.
- The 2D control branch slightly beats the 3D control branch on clip score and frame consistency, with the 3D branch winning only on pose accuracy.
Limitations
- Flickering appears in videos with sparse conditions such as pose.
- Pose accuracy and conditional consistency are evaluated only for pose; other conditions lack quantitative conditional metrics.
- Tune-A-Video comparison requires per-video fine-tuning, and the evaluation uses a single 120-video in-house set.
Experimental setup
Datasets: 120 reference videos for condition extraction
Benchmarks: in-house pose/canny/depth/segment condition set (120 videos)
Metrics: frame consistency clip score pose accuracy
Baselines: Tune-A-Video ControlNet Text2Video-Zero Sparse-Causal attention SC-Attn
Builds on: ControlNet Stable Diffusion Tune-A-Video DDIM inversion
Closest papers by meaning
- Generative Inbetweening through Frame-wise Conditions-Driven Video Generation (CVPR 2025 · similarity 0.89)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.89)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.88)
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation (CVPR 2024 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.