Research map / Video / Video generation
VidTwin: Video VAE with Decoupled Structure and Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VidTwin, a compact video VAE that decouples video into Structure latents (Q-Former over time plus spatial downsampling) and Dynamics latents (spatially averaged vectors), achieving 0.20% compression with competitive reconstruction and a simple DiT generation test on UCF-101.
Problem
Video autoencoders for latent video generation must compress heavily while preserving fine details and rapid motion; existing uniform-latent and content/motion decoupling methods either lose detail or are incompatible with generative models.
Approach
Encode video into two latents: a Structure latent produced by applying a Q-Former along the temporal axis independently per spatial location (followed by spatial downsampling), and a Dynamics latent produced by spatially downsampling then averaging over height and width, with both decoded jointly by summing their upsampled features.
Key results
- VidTwin reaches 0.20% compression rate with PSNR 28.137 and SSIM 0.8044 on MCL-JCV, versus CV-VAE PSNR 28.063 at 0.53%.
- On UCF-101 class-conditional generation, VidTwin attains FVD 193 versus 58 for MAGVIT-v2, 275 for Video-LaViT and 332 for TATS.
- Downstream DiT FLOPs are reported as 4 to 8 times smaller and training memory 2 to 3 times smaller than baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VidTwin instead of CV-VAE | video autoencoder reconstruction · MCL-JCV | PSNR: 28.063 -> 28.137 | Helped |
| use VidTwin instead of EMU-3 | video autoencoder reconstruction · MCL-JCV | FVD: 353.71 -> 388.86 | Hurt |
| use VidTwin instead of MAGVIT-v2 | class-conditioned video generation · UCF-101 | FVD: 58 -> 193 | Hurt |
| add Structure–Dynamics disentanglement rather than using a single latent (ablation) | VidTwin · MCL-JCV ablation | PSNR: 23.512 -> 26.116 | Helped |
| add Dynamics Latent spatial averaging instead of a Spatial Q-Former (ablation) | VidTwin · MCL-JCV ablation | PSNR: 24.835 -> 26.116 | Helped |
| add a Q-Former for Structure Latent extraction instead of convolution layers and an MLP (ablation) | VidTwin · MCL-JCV ablation | PSNR: 25.386 -> 26.116 | Helped |
| move spatial dimensions into the batch dimension for Structure Latent extraction (ablation) | VidTwin · MCL-JCV ablation | PSNR: 23.169 -> 26.116 | Helped |
Surprising findings
- Averaging dynamics latents spatially outperformed a spatial Q-Former for compressing the dynamics branch, which the authors attribute to disrupted spatial arrangement.
Limitations
- Generation evaluation is a simple baseline on UCF-101 and is not competitive with MAGVIT-v2 (FVD 58).
- Separating structure and dynamics latents introduces information loss, so isolated latent decodings are only illustrative.
- Human evaluation uses 15 evaluators and 20 samples each, a small-scale assessment.
Experimental setup
Datasets: self-collected 10M text-video dataset UCF-101
Benchmarks: MCL-JCV UCF-101
Metrics: PSNR SSIM LPIPS FVD compression rate MOS semantic/temporal/detail FLOPs training memory
Baselines: iVideoGPT MAGVIT-v2 CMD EMU-3 CV-VAE TATS Video-LaViT
Builds on: BLIP-2 Q-Former VQ-GAN CMD DiT Spatial-Temporal Transformer
Closest papers by meaning
- CV-VAE: A Compatible Video VAE for Latent Generative Video Models (NeurIPS 2024 · similarity 0.87)
- WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model (CVPR 2025 · similarity 0.87)
- Improved Video VAE for Latent Video Diffusion Model (CVPR 2025 · similarity 0.86)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.86)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.86)
- Improving the Diffusability of Autoencoders (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.