Research map / Video / Video generation
One-Minute Video Generation with Test-Time Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds TTT-MLP layers (neural-network hidden states updated at test time) to a pre-trained CogVideoX-5B diffusion transformer and fine-tunes it to generate one-minute videos from text storyboards on a Tom and Jerry dataset.
Problem
Transformers' self-attention cost grows quadratically with context, and linear RNN layers (Mamba 2, Gated DeltaNet) produce less coherent long videos because their matrix hidden states are small, so one-minute multi-scene video generation remains hard.
Approach
Replace the matrix hidden state of RNN layers with a two-layer MLP updated by a self-supervised gradient step during the forward pass (TTT-MLP), insert these layers with learned gates into a pre-trained Diffusion Transformer, and use local attention over 3-second segments plus global TTT layers with bi-directional scanning.
Key results
- TTT-MLP averages 1033 Elo on one-minute human evaluation, +34 over the best baselines (Sliding Window 999, Gated DeltaNet 991, Mamba 2 978).
- TTT-MLP gains +38 Elo on scene consistency and +39 on motion smoothness over the strongest baseline, with 100 videos per method.
- The 7.2B-parameter modified model is trained in about 50 hours on 256 H100 GPUs with preliminary systems optimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add gated, bidirectional TTT-MLP layers | CogVideo-X 5B · One-minute video generation | Average Elo: +34 ELO points | Helped |
| Add gated, bidirectional TTT-MLP layers | CogVideo-X 5B · One-minute video generation | Scene consistency Elo: +38 | Helped |
| Add gated, bidirectional TTT-MLP layers | CogVideo-X 5B · One-minute video generation | Motion smoothness Elo: +39 | Helped |
| Add gated, bidirectional TTT-MLP layers | CogVideo-X 5B · One-minute video generation | Aesthetics Elo: +31 | Helped |
| Add gated, bidirectional TTT-MLP layers | CogVideo-X 5B · One-minute video generation | Text alignment Elo | No clear effect |
Surprising findings
- Text-following ability showed no significant differences among methods, since all methods share the same dense segment prompts and local attention.
Limitations
- Limited physical realism in generated videos.
- Implementation efficiency can be further improved.
- Evaluation restricted to a single cartoon domain (Tom and Jerry) with proof-of-concept scope.
Experimental setup
Datasets: Tom and Jerry cartoon episodes (81 episodes, ~7 hours, human-annotated)
Benchmarks: Tom and Jerry one-minute and 18-second human evaluation
Metrics: Elo rating text alignment motion smoothness aesthetics scene consistency
Baselines: Local attention TTT-Linear Mamba 2 Gated DeltaNet Sliding-window attention
Builds on: Test-Time Training layers CogVideoX Mamba 2 Gated DeltaNet Flash Attention
Closest papers by meaning
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.85)
- Learning to (Learn at Test Time): RNNs with Expressive Hidden States (ICML 2025 · similarity 0.84)
- LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.84)
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · similarity 0.84)
- Masked Generative Nested Transformers with Decode Time Scaling (ICML 2025 · similarity 0.83)
- DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.