Research map / Image generation / Autoregressive and latent generation
Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DC-AE, a high spatial-compression autoencoder (up to f128) using residual autoencoding and decoupled three-phase high-resolution adaptation, which speeds up latent diffusion transformers while matching or improving FID.
Problem
High spatial-compression autoencoders (e.g., f32–f128) for latent diffusion suffer large reconstruction accuracy drops and generalization penalties when moving from low to high resolution, so diffusion models still rely on f8 autoencoders with patch embedding.
Approach
Residual autoencoding adds non-parametric space-to-channel shortcuts in downsample/upsample and middle blocks so modules learn residuals over a space-to-channel mapping; decoupled high-resolution adaptation uses a three-phase schedule (low-res full training, high-res latent adaptation of middle layers, low-res local GAN refinement of decoder head) to avoid high-res GAN training.
Key results
- DC-AE-f64p1 on UViT-H ImageNet 512x512 gives 19.1x inference throughput and 17.9x training throughput over SD-VAE-f8p2 while improving FID from 3.55 to 3.01 with CFG.
- On ImageNet 512x512 reconstruction, DC-AE improves rFID for f64c128 from 16.84 (SD-VAE) to 0.22 and for f128c512 from 100.74 to 0.23.
- DC-AE-f32+USiT-2B reaches 1.72 FID with CFG on ImageNet 512x512, outperforming EDM2-XXL (1.81) and MAR-L (1.73 is close; MAGVIT-v2 1.91).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SD-VAE with DC-AE at f32c32 | Image reconstruction · ImageNet 256×256 f32c32 | rFID: 2.64 -> 0.69 | Helped |
| replace SD-VAE with DC-AE at f64c128 | Image reconstruction · ImageNet 512×512 f64c128 | rFID: 16.84 to 0.22 | Helped |
| replace SD-VAE with DC-AE at f128c512 | Image reconstruction · ImageNet 512×512 f128c512 | rFID: 100.74 to 0.23 | Helped |
| use DC-AE-f64p1 instead of SD-VAE-f8p2 | UViT-H · ImageNet 512×512 | FID w/ CFG: 3.55 -> 3.01 | Helped |
| use DC-AE-f64p1 instead of SD-VAE-f8p2 | UViT-H · ImageNet 512×512 | Inference throughput: 19.1× | Helped |
| use DC-AE-f32p1 instead of SD-VAE-f8p2 | DiT-XL · ImageNet 512×512 | FID w/ CFG: 3.04 -> 2.84 | Helped |
| use DC-AE-f32p1 instead of SD-VAE-f8p2 | PIXART-α · MJHQ 512×512 | FID: 6.3 -> 6.1 | Helped |
| shift spatial compression from the diffusion model to the autoencoder (ablation) | UViT-S · ImageNet 512×512, 64 tokens | FID w/ CFG: 95.93 -> 35.96 | Helped |
| directly reach f64 compression with DC-AE rather than combining lower compression with patch compression (ablation) | UViT-S · ImageNet 512×512, 64 tokens | FID w/ CFG: 76.57 -> 35.96 | Helped |
| add Residual Autoencoding shortcuts (ablation) | High spatial-compression DC-AE · ImageNet 256×256 | rFID | Helped |
| add high-resolution latent adaptation by tuning middle layers (ablation) | High spatial-compression DC-AE · ImageNet 1024×1024 | memory cost: 153.98 GB → 67.81 GB | Helped |
| use DC-AE-f64 rather than SD-VAE-f8 | UViT-S · ImageNet 512×512 | FID | Hurt |
Surprising findings
- Adding extra encoder/decoder stages with SD-VAE blocks to high-compression autoencoders performs worse than a simple space-to-channel operation, indicating an optimization difficulty.
- DC-AE-f64p1 underperforms SD-VAE-f8p2 on UViT-S but outperforms it on larger UViT-2B, suggesting benefits emerge with model scale.
Limitations
- Higher-compression autoencoders, with more latent channels, may need more diffusion model capacity; DC-AE-f64 underperforms on small UViT-S.
- High-resolution latent adaptation and GAN training still incur extra training cost.
Experimental setup
Datasets: ImageNet SAM MapillaryVistas FFHQ MJHQ-30K
Benchmarks: ImageNet 256x256 reconstruction ImageNet 512x512 FFHQ 1024x1024 MapillaryVistas 2048x2048 MJHQ-30K 512x512
Metrics: rFID PSNR SSIM LPIPS FID CLIP Score Inception Score Precision Recall CMMD
Baselines: SD-VAE-f8 Flux-VAE-f8 SD3-VAE-f8 SDXL-VAE-f8 Asym-VAE-f8 SD-VAE-f16 SD-VAE-f32 MAGVIT-v2 EDM2-XXL MAR-L
Builds on: latent diffusion models SD-VAE EfficientViT UViT DiT
Closest papers by meaning
- DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence (NeurIPS 2026 · similarity 0.92)
- LiteVAE: Lightweight and Efficient Variational Autoencoders for Latent Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.88)
- PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher (NeurIPS 2024 · similarity 0.88)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.