Research map / Video / Video generation
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TF-T2V, a two-branch video diffusion framework that learns text-to-video generation from text-free videos (plus image-text data), with a temporal coherence loss, and shows scaling gains from adding unlabeled videos.
Problem
Text-to-video diffusion models are limited by the scarce and costly video-text pairs (e.g., WebVid10M), while far more unlabeled video is available; the scaling trend of video generation is under-explored.
Approach
Separate spatial appearance learning (content branch, trained on image-text pairs) from motion learning (motion branch, image-conditioned video generation on text-free videos) within one unified 3D-UNet with shared weights, plus a frame-difference temporal coherence loss.
Key results
- Doubling training data with text-free videos improves TF-T2V FID from 9.67 to 8.19 and FVD from 484 to 441 on MSR-VTT.
- Semi-supervised TF-T2V-Semi reaches FID 7.64 and FVD 366, beating ModelScopeT2V (FID 11.09, FVD 550).
- On compositional synthesis, TF-T2V depth error 0.209 vs VideoComposer 0.217 with text and depth; sketch error 0.1146 vs 0.1161.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Internal10M text-free videos to WebVid10M training | TF-T2V · MSR-VTT | FID: 9.67 -> 8.19 | Helped |
| Add Internal10M text-free videos to WebVid10M training | TF-T2V · MSR-VTT | FVD: 484 -> 441 | Helped |
| Add Internal10M text-free videos to WebVid10M training | TF-T2V · MSR-VTT | CLIPSIM: 0.2953 -> 0.2991 | Helped |
| Add labeled WebVid10M video-text data to text-free Internal10M training | TF-T2V · MSR-VTT | FID: 8.19 -> 7.64 | Helped |
| Add labeled WebVid10M video-text data to text-free Internal10M training | TF-T2V · MSR-VTT | FVD: 441 -> 366 | Helped |
| Add labeled WebVid10M video-text data to text-free Internal10M training | TF-T2V · MSR-VTT | CLIPSIM: 0.2991 -> 0.3032 | Helped |
| Add temporal coherence loss (ablation) | TF-T2V · Text-to-video generation | Frame consistency: 89.71 -> 91.06 | Helped |
| Use TF-T2V instead of VideoComposer with text and depth conditions | VideoComposer · Depth control | Depth error: 0.217 -> 0.209 | Helped |
| Use TF-T2V instead of VideoComposer with text and sketch conditions | VideoComposer · Sketch control | Sketch error: 0.1161 -> 0.1146 | Helped |
| Use TF-T2V instead of VideoComposer with text and motion-vector conditions | VideoComposer · Motion-vector control | EPE: 1.98 -> 1.88 | Helped |
Limitations
- Scaling evidence relies on a single proprietary internal 10M video set that is not released
- Human evaluation uses only 100 videos
- Gains over ModelScopeT2V on CLIPSIM are small or absent in the single-model setting
Experimental setup
Datasets: WebVid10M Internal10M LAION-5B MSR-VTT
Benchmarks: MSR-VTT
Metrics: FID FVD CLIPSIM depth error sketch error EPE frame consistency human preference
Baselines: ModelScopeT2V VideoComposer Text2video-Zero ZeroScope Make-A-Video CogVideo VideoLDM Control-A-Video
Builds on: ModelScopeT2V VideoComposer Latent diffusion models CLIP DDIM
Closest papers by meaning
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.91)
- T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation (CVPR 2025 · similarity 0.90)
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.90)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.90)
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.90)
- Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.