Research map / Video / Video generation
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiGen, a diffusion T2V method that decouples generation structurally (spatial then temporal reasoning with a unified 3D-UNet) and at the content level (motion and appearance guidance factors), fine-tuned from Stable Diffusion/ModelScopeT2V.
Problem
Text-to-video diffusion models intertwine spatial content and temporal dynamics, making T2V hard; methods either have good spatial quality with little motion or dynamic motion with poor spatial quality.
Approach
Decouple T2V hierarchically: at the structure level, generate a spatial prior from text (spatial reasoning) and then temporal motion conditioned on that prior (temporal reasoning) with a shared unified denoiser; at the content level, condition on a motion factor (inter-frame pixel differences) and an appearance factor (semantic similarity via DINO/CLIP between frames) extracted from videos.
Key results
- On MSR-VTT, HiGen reports FID 8.60 and FVD 406 versus ModelScopeT2V's 11.09 and 550, and CLIPSIM 0.2947 vs 0.2930.
- In human evaluation, HiGen scores 74.0% on temporal quality, versus 55.2% for VideoCrafter.
- Ablation shows structure-level decoupling raises CLIPSIM but lowers temporal consistency, and adding content-level decoupling recovers it (0.944 temporal consistency, 0.318 CLIPSIM).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiGen instead of ModelScopeT2V | Text-to-video generation · MSR-VTT | FID: 11.09 -> 8.60 | Helped |
| Use HiGen instead of ModelScopeT2V | Text-to-video generation · MSR-VTT | FVD: 550 -> 406 | Helped |
| Use HiGen instead of ModelScopeT2V | Text-to-video generation · MSR-VTT | CLIPSIM: 0.2930 -> 0.2947 | Helped |
| Use HiGen instead of VideoCrafter | Text-to-video generation · Human evaluation | temporal quality preference: 55.2% to 74.0% | Helped |
| Add structure-level decoupling (ablation) | HiGen · Ablation prompts | CLIPSIM: 0.292 -> 0.313 | Helped |
| Add structure-level decoupling (ablation) | HiGen · Ablation prompts | Temporal Consistency: 0.931 -> 0.889 | Hurt |
| Add content-level decoupling (ablation) | HiGen with structure-level decoupling · Ablation prompts | Temporal Consistency: 0.889 -> 0.944 | Helped |
| Add content-level decoupling (ablation) | HiGen with structure-level decoupling · Ablation prompts | CLIPSIM: 0.313 -> 0.318 | Helped |
| Increase the motion factor (ablation) | HiGen · Motion factor sensitivity analysis | Temporal Consistency | Hurt |
| Increase the appearance factor (ablation) | HiGen · Appearance factor sensitivity analysis | Temporal Consistency | Hurt |
| Vary FPS instead of using content-level decoupling (ablation) | HiGen · FPS sensitivity analysis | temporal dynamics | No clear effect |
Surprising findings
- Lower temporal consistency scores were associated with livelier, more dynamic videos, suggesting over-prioritizing temporal consistency can limit vibrancy.
Limitations
- Object detail generation lags behind current image synthesis models due to compute and data quality limits.
- Struggles to model human and animal actions that follow common sense, especially with large motion.
- FID/FVD metrics may not accurately reflect generated video quality, per the authors.
Experimental setup
Datasets: internal watermark-free video-text pairs (17M) LAION-400M MSR-VTT
Benchmarks: MSR-VTT
Metrics: FID FVD CLIPSIM Temporal Consistency human evaluation (visual quality, temporal quality, text ali
Baselines: ModelScopeT2V Gen-2 Text2Video-Zero VideoCrafter CogVideo Latent-Shift Make-A-Video Video LDM MagicVideo VideoComposer PYoCo
Builds on: Stable Diffusion ModelScopeT2V VideoFusion DINO CLIP
Closest papers by meaning
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.89)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.89)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.89)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.89)
- ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Way (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.