Research map / Video / Video generation
CV-VAE: A Compatible Video VAE for Latent Generative Video Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a continuous 3D video VAE, inflated from the Stable Diffusion image VAE with a latent-space regularization against the frozen 2D decoder, so existing image/video diffusion models can use 4x temporal compression with minimal finetuning.
Problem
Latent video diffusion models use 2D image VAEs with uniform frame sampling for temporal compression, producing unsmooth motion; training a new 3D video VAE creates a latent-space gap with pretrained T2I/video models that is costly to bridge.
Approach
Inflate the SD image VAE into a 3D VAE initialized from its weights and add a latent regularization loss that requires the video VAE's latents to be decoded correctly by the frozen 2D image decoder, aligning the latent space with pretrained models; also uses a half-2D/half-3D conv architecture to reduce cost.
Key results
- CV-VAE (2D+3D) reaches 27.6 PSNR on COCO-Val and 28.5 PSNR on Webvid-Val at 4x frame compression, matching or beating VAE-SD2.1 image reconstruction and outperforming VAE-OSP on video.
- SD2.1 with CV-VAE swapped in without finetuning gives FID 57.6 vs 57.3 for the original SD2.1 on COCO-Val, showing latent compatibility.
- SVD+CV-VAE-V with output-layer finetuning (4x frames, 97 frames) improves UCF-101 FVD from 681 to... and MSR-VTT FVD from 310 (SVD) to 295 versus SVD+RIFE 253 with lower KVD 2.26 vs 3.12.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the SD image VAE with CV-VAE (2D+3D) | Image and video reconstruction · COCO-Val | PSNR: 26.6 -> 27.6 | Helped |
| Replace the SD image VAE with CV-VAE (2D+3D) | Image and video reconstruction · Webvid-Val | SSIM: 0.810 -> 0.817 | Helped |
| Replace the SD image VAE with CV-VAE (2D+3D) | Image and video reconstruction · Webvid-Val | PSNR: 28.9 -> 28.5 | Hurt |
| Use 2D decoder regularization instead of 2D encoder regularization (ablation) | CV-VAE · Webvid-Val | PSNR: 26.0 -> 28.0 | Helped |
| Add both 2D encoder and decoder regularization instead of decoder regularization alone (ablation) | CV-VAE · Webvid-Val | PSNR: 28.0 -> 27.6 | Hurt |
| Use first-frame mapping rather than random-frame mapping (ablation) | CV-VAE regularization · Webvid-Val | PSNR: 28.4 -> 26.6 | Hurt |
| Use random-frame mapping rather than average mapping (ablation) | CV-VAE regularization · Webvid-Val | PSNR: 27.9 -> 28.4 | Helped |
| Replace the original SVD VAE with CV-VAE in image mode | SVD · UCF-101 | FVD: 402 -> 419 | Hurt |
| Replace the original SVD VAE with CV-VAE in image mode | SVD · MSR-VTT | FVD: 310 -> 262 | Helped |
| Use CV-VAE video mode and fine-tune the SVD output layer | SVD · UCF-101 | PIC: 0.791 -> 0.858 | Helped |
| Use CV-VAE video mode and fine-tune the SVD output layer | SVD · MSR-VTT | PIC: 0.588 -> 0.734 | Helped |
| Replace the original VAE with CV-VAE in SD2.1 without fine-tuning | SD2.1 text-to-image generation · COCO2017-Val | FID: 57.3 -> 57.6 | No clear effect |
Surprising findings
- Regularizing with the frozen 2D decoder works better than regularizing with the 2D encoder (Webvid PSNR 28.0 vs 26.0).
- The 3D-only variant gives only marginal gains over the 2D+3D variant while costing about 30% more parameters and compute.
Limitations
- Latent channel dimension is fixed at 4 to match the SD image VAE, limiting reconstruction quality.
- Slight color discrepancy remains between CV-VAE and the original image VAE latent space.
- Video reconstruction is worse than VAE-SD3 when using 16-channel latents, due to higher temporal compression.
Experimental setup
Datasets: LAION-COCO Unsplash WebVid-10M COCO2017 UCF-101 MSR-VTT in-house video data
Benchmarks: COCO2017 val WebVid val UCF-101 MSR-VTT
Metrics: PSNR SSIM LPIPS FID CLIP score PIC FVD KVD
Baselines: VAE-SD2.1 VQGAN TATS VAE-OSP (Open-Sora-Plan VAE) FIRE / RIFE interpolation SVD VideoCrafter2
Builds on: Stable Diffusion VAE latent diffusion models VQGAN SVD VideoCrafter2
Closest papers by meaning
- Improved Video VAE for Latent Video Diffusion Model (CVPR 2025 · similarity 0.94)
- WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model (CVPR 2025 · similarity 0.92)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.88)
- EQ-VAE: Equivariance Regularized Latent Space for Improved Generative Image Modeling (ICML 2025 · similarity 0.88)
- VidTwin: Video VAE with Decoupled Structure and Dynamics (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.