Research map / Image generation / Autoregressive and latent generation
LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LiteVAE, a wavelet-based (Haar DWT) autoencoder for latent diffusion that matches standard VAE reconstruction quality with about six-fold fewer encoder parameters, plus decoder SMC and training tweaks that improve rFID.
Problem
Standard VAEs used in latent diffusion models are computationally expensive to train and to run as encoders, and the autoencoder design space is underexplored.
Approach
Apply a multi-level Haar discrete wavelet transform to the input image, extract multiscale features from the wavelet sub-bands with lightweight UNets, aggregate them into the latent, and decode with the standard SD-VAE decoder; also replace decoder group normalization with self-modulated convolution and use lower-resolution pretraining, a UNet discriminator, and high-frequency losses.
Key results
- Base LiteVAE uses about 6.75M encoder parameters versus 34.16M for the SD-VAE encoder while matching its reconstruction quality.
- LiteVAE-B roughly doubles encoder throughput and requires considerably less GPU memory than the SD-VAE encoder.
- On ImageNet 128x128 ablations, a UNet discriminator gives rFID 1.01 versus 1.61 for PatchGAN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a learned LiteVAE latent instead of a fixed DWT representation | FFHQ latent diffusion · FFHQ (256×256) | FID: 12.51 -> 8.03 | Helped |
| Use a fixed DWT encoder with n_z=12 instead of SD-VAE | Autoencoder · FFHQ | rFID: 0.85 -> 0.70 | Helped |
| Use a fixed DWT encoder with n_z=12 instead of SD-VAE | Autoencoder · DeepFashion | rFID: 1.64 -> 1.71 | Hurt |
| Use a fixed DWT encoder with n_z=12 instead of SD-VAE | Autoencoder · ImageNet | rFID: 0.80 -> 1.65 | Hurt |
| Increase fixed DWT latent size from n_z=12 to n_z=48 (ablation) | DWT encoder · ImageNet | rFID: 1.65 -> 0.32 | Helped |
| Replace the PatchGAN discriminator with a UNet discriminator (ablation) | LiteVAE training · Reconstruction | rFID: 1.61 -> 1.01 | Helped |
| Replace the PatchGAN discriminator with a UNet discriminator (ablation) | LiteVAE training · Reconstruction | LPIPS: 0.063 -> 0.070 | Hurt |
| Replace the PatchGAN discriminator with a UNet discriminator (ablation) | LiteVAE training · Reconstruction | PSNR: 29.61 -> 29.24 | Hurt |
| Replace the PatchGAN discriminator with a UNet discriminator (ablation) | LiteVAE training · Reconstruction | SSIM: 0.87 -> 0.86 | No clear effect |
| Remove the adaptive adversarial-loss weight and use a constant weight (ablation) | LiteVAE training · ImageNet 128×128 | reconstruction quality | Helped |
| Add wavelet and Gaussian high-frequency reconstruction losses (ablation) | LiteVAE training · ImageNet 128×128 | reconstruction metrics | Helped |
| Replace group normalization with self-modulated convolution in the decoder (ablation) | LiteVAE decoder · Reconstruction | reconstruction quality | Helped |
Surprising findings
- A non-learned fixed Haar wavelet encoder (no encoder parameters) reaches rFID comparable to a learned SD-VAE encoder on FFHQ (0.70 vs 0.85).
- Removing the 1x1 convolutions around the latent space improves reconstruction quality by enforcing less KL regularization.
Limitations
- The non-learned wavelet encoder falls short of SD-VAE quality on the more diverse ImageNet dataset.
- Efficiency benefits beyond natural images (other domains, tokenization) are left to future work.
Experimental setup
Datasets: ImageNet FFHQ CelebA-HQ DeepFashion
Benchmarks: ImageNet FFHQ CelebA-HQ DeepFashion
Metrics: rFID LPIPS PSNR SSIM FID MMD
Baselines: Stable Diffusion VAE PatchGAN discriminator StyleGAN discriminator
Builds on: latent diffusion models Stable Diffusion VAE ADM UNet modulated convolution (StyleGAN2)
Closest papers by meaning
- Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models (ICLR 2025 · similarity 0.89)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.88)
- DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence (NeurIPS 2026 · similarity 0.88)
- Improving the Diffusability of Autoencoders (ICML 2025 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.