Research map / Video / Video generation
Goku: Flow Based Video Generative Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Goku, a family of 2B and 8B rectified-flow transformers for joint image and video generation, with a curated 160M image-text and 36M video-text data pipeline and large-scale training infrastructure.
Problem
Building industry-grade joint image-and-video generative models that are trained efficiently at scale on large, curated image and video data.
Approach
Combines a jointly trained image-video 3D VAE, full attention over packed image and video tokens, 3D RoPE, QK-norm, and rectified-flow training in a GenTron-style transformer, together with a multi-stage data curation pipeline and MegaScale-style training infrastructure.
Key results
- Goku-T2I (2B) reaches 0.76 GenEval with prompt rewriting (0.70 without) and 83.65 on DPG-Bench.
- Goku-T2V reaches 84.85 overall on VBench, reported as top of the leaderboard as of 2025-01-25.
- On zero-shot UCF-101 at 128x128, Goku-2B achieves FVD 217.24, lower than the compared methods in Table 5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use rectified flow instead of DDPM | Goku-1B · ImageNet-1K 256→256, 200k training steps | FID-50K: 3.0795 -> 2.7472 | Helped |
| use rectified flow instead of DDPM | Goku-1B · ImageNet-1K 256→256, 400k training steps | FID-50K: 2.5231 -> 2.1572 | Helped |
| add prompt rewriting | Goku-T2I (2B) · GenEval | score: 0.70 -> 0.76 | Helped |
| add prompt rewriting | Goku-T2I (2B) · T2I-CompBench | Color, Shape, and Texture: 0.7521 -> 0.7561; 0.4832 -> 0.5759; 0.6691 -> 0.7071 | Helped |
| scale the model from 2B to 8B parameters (ablation) | Goku-T2V · Text-to-video generation | visual structure quality | Helped |
| remove joint image-and-video training (ablation) | Goku-T2V (8B) · 480p video generation | frame quality | Hurt |
Surprising findings
- Rectified flow converged faster than DDPM on ImageNet class-conditional generation (RF at 400k steps matched DDPM at roughly 1000k steps in FID).
- Goku-T2I improves GenEval from 0.70 to 0.76 when prompts are rewritten with ChatGPT-4o into longer descriptions.
Limitations
- Training data is largely proprietary and internal, limiting reproducibility.
- Some benchmark comparisons rely on prompt rewriting and self-reported leaderboard results.
Experimental setup
Datasets: LAION Panda-70M InternVid OpenVid-1M Pexels ImageNet-1K UCF-101 internal proprietary image and video data
Benchmarks: GenEval T2I-CompBench DPG-Bench UCF-101 VBench
Metrics: GenEval score T2I-CompBench attribute scores DPG-Bench average FVD IS FID VBench total score
Baselines: SD3 DALL-E 3 Emu 3 PixArt-α SDXL CogVideoX-5B HunyuanVideo Kling Gen-3 Luma CausVid Make-A-Video
Builds on: rectified flow GenTron NaViT Ulysses sequence parallelism FSDP
Closest papers by meaning
- Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis (CVPR 2024 · similarity 0.87)
- EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (CVPR 2024 · similarity 0.87)
- GenAI Arena: An Open Evaluation Platform for Generative Models (NeurIPS 2024 · similarity 0.86)
- Mean Flows for One-step Generative Modeling (NeurIPS 2025 · similarity 0.86)
- Contextual Flow Matching: Adaptive Step Selection in Flow Models for Efficient Visual Generation (NeurIPS 2026 · similarity 0.86)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.