Research map / Video / Video generation
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CogVideoX, a large-scale diffusion transformer text-to-video model (2B and 5B) with a 3D causal VAE, expert transformer with expert adaptive LayerNorm, 3D full attention, progressive multi-resolution training, and a dense video recaptioning pipeline.
Problem
Text-to-video generation models struggle with limited motion, short durations, and coherent narratives that follow text prompts, and open models lag closed ones in quality.
Approach
Combines a 3D causal VAE (8x8x4 compression) with a DiT that uses expert adaptive LayerNorm for separate text/vision modulation, 3D full attention instead of separated spatial-temporal attention, Multi-Resolution Frame Pack for mixed-duration/resolution batches, and a dense-caption data pipeline (CogVLM2-Caption) to improve text-video alignment.
Key results
- CogVideoX-5B achieves best VBench Multiple Objects (70.95) and Dynamic Quality (69.5) and GPT4o-MT Score 3.36, best on five of seven metrics.
- In human evaluation against Kling (2024.7), CogVideoX-5B total score 2.74 vs 2.17.
- The 3D VAE achieves PSNR 29.1 and flickering 85.5 versus Open-Sora (28.5, 92.4) and Open-Sora-Plan (27.6, 90.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Human Action | Human Action: 96.4 -> 96.8 | Helped |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Scene | Scene: 55.58 -> 55.44 | Hurt |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Dynamic Degree | Dynamic Degree: 49.17 -> 62.22 | Helped |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Multiple Objects | Multiple Objects: 64.88 -> 70.95 | Helped |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Appearance Style | Appearance Style: 25.13 -> 24.44 | Hurt |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench Dynamic Quality | Dynamic Quality: 63.7 -> 69.5 | Helped |
| Use CogVideoX-5B instead of the strongest reported baseline | Text-to-video generation · VBench GPT4o-MT Score | GPT4o-MT Score: 2.68 -> 3.36 | Helped |
| Use CogVideoX-5B instead of Kling | Video generation · Human evaluation | Total Score: 2.17 -> 2.74 | Helped |
| Use CogVideoX 3D VAE instead of Open-Sora-Plan VAE | Video reconstruction · WebVid validation, 256×256, 17 frames | Flickering: 90.2 -> 85.5 | Helped |
| Use CogVideoX 3D VAE instead of Open-Sora VAE | Video reconstruction · WebVid validation, 256×256, 17 frames | PSNR: 28.5 -> 29.1 | Helped |
| Increase latent channels from 4 to 16 while using 8×8×4 compression (ablation) | 3D VAE · 3D VAE ablation | PSNR: 28.4 -> 28.7 | Helped |
| Increase latent channels from 16 to 32 while using 8×8×4 compression (ablation) | 3D VAE · 3D VAE ablation | PSNR: 28.7 -> 30.5 | Helped |
Surprising findings
- 2D+1D attention was unstable and prone to collapse at larger model size, while 3D full attention trained stably.
- Increasing VAE compression to 16x16x8 made model convergence extremely difficult despite larger latent channels.
Limitations
- Larger VAE compression ratios (16x16x8) were hard to converge; left for future work.
- High-quality fine-tuning slightly degraded semantic ability of the model.
- 3D full attention substantially increases inference time at high resolution versus 2D+1D attention.
Experimental setup
Datasets: WebVid Panda70M LAION-5B COYO-700M COCO Caption
Benchmarks: VBench Dynamic Quality GPT4o-MTScore WebVid validation set Human evaluation (100 prompts vs Kling)
Metrics: Human Action Scene Dynamic Degree Multiple Objects Appearance Style Dynamic Quality GPT4o-MT Score FVD CLIP4Clip score PSNR
Baselines: T2V-Turbo AnimateDiff VideoCrafter-2.0 OpenSora V1.2 Show-1 Gen-2 Pika LaVie-2 Kling MMDiT
Builds on: DiT Patch'n Pack VDM v-prediction RoPE Video-LLaMA
Closest papers by meaning
- Grid Diffusion Models for Text-to-Video Generation (CVPR 2024 · similarity 0.89)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.89)
- LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.89)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.88)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.88)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.