Research map / Video / Video generation
Improved Video VAE for Latent Video Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IV-VAE, a video VAE with a keyframe-based temporal compression (KTC) dual-branch architecture and group causal convolution (GCConv), plus a MotionHD benchmark, for better 4x temporal compression reconstruction and latent video diffusion.
Problem
Existing video VAEs inflate pre-trained image VAEs with the same latent channel count into 3D causal structures, which yields poor initialization for temporal compression and unbalanced reconstruction quality across frames within a frame group.
Approach
Initialize a dual-branch video VAE from a lower-dimensional (Z/2) image VAE for keyframes while a 3D group causal convolution branch handles temporal motion, and replace plain causal convolution with group causal convolution so frames within a group interact bidirectionally while keeping causality across groups.
Key results
- On Kinetics-600 at Z=4, IV-VAE (107M) reaches FVD 8.01 and PSNR 34.29 versus OD-VAE's 10.69 FVD and 33.88 PSNR with less than half the parameters.
- On MotionHD at 1080P with Z=16, IV-VAE achieves PSNR 40.33 versus CogX-VAE's 38.68 (about 1.65 dB gain).
- On video generation with Latte-XL, IV-VAE (Z=4) achieves FVD 118.6 on SkyTimelapse versus CV-VAE's 128.0 and 541.6 on Kinetics-400 versus CV-VAE's 552.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IV-VAE instead of CogX-VAE | Video VAE reconstruction · Kinetics-600 | FVD: 3.17 -> 2.97 | Helped |
| Use IV-VAE instead of CogX-VAE | Video VAE reconstruction · ActivityNet | PSNR: 40.68 -> 42.61 | Helped |
| Use IV-VAE instead of CogX-VAE | Video VAE reconstruction · MotionHD 1080P | PSNR: 38.68 -> 40.33 | Helped |
| Use IV-VAE instead of OD-VAE | Latte video generation · Kinetics-400 | FVD: 548.7 -> 541.6 | Helped |
| Use IV-VAE instead of CV-VAE | Latte video generation · SkyTimelapse | FVD: 128.0 -> 118.6 | Helped |
| Add group causal convolution (ablation) | Causal VAE · Kinetics-600 256×256 | PSNR: 31.29 -> 31.64 | Helped |
| Add group causal convolution (ablation) | Causal VAE · Kinetics-600 512×512 | PSNR: 34.88 -> 35.21 | Helped |
| Add keyframe-based temporal compression (ablation) | Causal VAE · Kinetics-600 256×256 | PSNR: 31.29 -> 31.86 | Helped |
| Add keyframe-based temporal compression (ablation) | Causal VAE · Kinetics-600 512×512 | PSNR: 34.88 -> 35.46 | Helped |
| Add temporal motion perception enhancement (ablation) | Causal VAE + GCConv + KTC · Kinetics-600 256×256 | PSNR: 32.12 -> 32.24 | Helped |
| Add temporal motion perception enhancement (ablation) | Causal VAE + GCConv + KTC · Kinetics-600 512×512 | PSNR: 35.69 -> 35.85 | Helped |
| Increase the basic feature dimension from 64 to 96 (ablation) | IV-VAE · Kinetics-600 | PSNR: 39.02 -> 39.57 | Helped |
Surprising findings
- An image VAE with half the latent channels provides enough spatial compression to initialize a video VAE with the full channel count, converging faster than same-dimension initialization.
Limitations
- Gains in PSNR at 480P and some metrics are modest or mixed relative to baselines (e.g., slightly lower PSNR than OS-VAE due to parameter limits).
Experimental setup
Datasets: Kinetics-600 ActivityNet OpenVid-0.4M Kinetics-400 SkyTimelapse
Benchmarks: Kinetics-600 ActivityNet MotionHD Kinetics-400 SkyTimelapse
Metrics: PSNR SSIM LPIPS FVD
Baselines: SVD-VAE CV-VAE OpenSora V1.2 VAE OD-VAE CogVideoX VAE Causal VAE baseline
Builds on: MagViT-v2 OD-VAE Latte Stable Diffusion VAE central initialization
Closest papers by meaning
- CV-VAE: A Compatible Video VAE for Latent Generative Video Models (NeurIPS 2024 · similarity 0.94)
- WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model (CVPR 2025 · similarity 0.91)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.87)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.87)
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.