Research map / Video / Video generation
WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WF-VAE, a video VAE that routes low-frequency Haar wavelet energy through a side pathway plus a Causal Cache for lossless block-wise inference, achieving higher throughput and lower memory than prior video VAEs.
Problem
Video VAEs are a computational bottleneck for latent video diffusion model training at high resolution and duration, and block-wise inference causes discontinuities/flickering in the latent space.
Approach
Use multi-level 3D/2D Haar wavelet transforms to create a main energy-flow pathway that carries low-frequency video energy into the latent (letting the backbone use fewer 3D convolutions), plus a Causal Cache that caches causal-conv tail frames so block-wise inference matches direct inference exactly.
Key results
- WF-VAE-L (16 channels) reaches 35.87 PSNR and 0.0175 LPIPS on Panda70M vs. CogVideoX VAE's 35.79 and 0.0198.
- WF-VAE-L encodes a 512x512 video with about 7170 MB memory vs. about 31944 MB for OD-VAE (445% higher), and encodes in 0.0513 s vs. 0.0945 s for OD-VAE.
- Causal Cache gives identical PSNR/LPIPS to direct inference, while tiling in OD-VAE drops PSNR by 1.80 dB and Allegro by 6.40 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · WebVid-10M | PSNR: 35.72 -> 35.76 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · WebVid-10M | SSIM: 0.9434 -> 0.9430 | Hurt |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · WebVid-10M | LPIPS: 0.0277 -> 0.0230 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · WebVid-10M | rFVD: 59.83 -> 54.36 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · Panda70M | PSNR: 35.79 -> 35.87 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · Panda70M | SSIM: 0.9527 -> 0.9538 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · Panda70M | LPIPS: 0.0198 -> 0.0175 | Helped |
| Use WF-VAE-L instead of CogVideoX-VAE | Video VAE reconstruction · Panda70M | rFVD: 43.23 -> 39.40 | Helped |
| Add low-frequency energy flow pathways at layers 2 and 3 (ablation) | WF-VAE · Kinetics-400 | PSNR: 27.94 -> 28.21 | Helped |
| Add low-frequency energy flow pathways at layers 2 and 3 (ablation) | WF-VAE · Kinetics-400 | LPIPS: 0.0737 -> 0.0690 | Helped |
| Add WL Loss (ablation) | WF-VAE · Kinetics-400 | PSNR: 27.90 -> 28.21 | Helped |
| Add WL Loss (ablation) | WF-VAE · Kinetics-400 | LPIPS: 0.0692 -> 0.0690 | No clear effect |
Surprising findings
- Conventional caching in CogVideoX degrades PSNR by 0.38 dB under block-wise inference, indicating that group normalization breaks lossless caching.
Limitations
- Decoder has more backbone parameters than the encoder, which the authors consider redundant.
- Larger latent channel counts (16) make diffusion training converge more slowly, limiting generation gains.
- Generation evaluation uses Latte-L on small datasets (UCF-101, SkyTimelapse) rather than large-scale LVDM training.
Experimental setup
Datasets: Kinetics-400 UCF-101 SkyTimelapse Panda70M WebVid-10M
Benchmarks: Kinetics-400 WebVid-10M Panda70M UCF-101 SkyTimelapse
Metrics: PSNR SSIM LPIPS rFVD FVD16 Inception Score encode/decode time memory
Baselines: SD-VAE SVD-VAE CV-VAE OD-VAE Open-Sora VAE Allegro VAE CogVideoX VAE
Builds on: LDM (latent diffusion) Latte OD-VAE Haar wavelet transform
Closest papers by meaning
- CV-VAE: A Compatible Video VAE for Latent Generative Video Models (NeurIPS 2024 · similarity 0.92)
- Improved Video VAE for Latent Video Diffusion Model (CVPR 2025 · similarity 0.91)
- Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding (ICML 2021 · similarity 0.88)
- VidTwin: Video VAE with Decoupled Structure and Dynamics (CVPR 2025 · similarity 0.87)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.86)
- LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.