Research map / Image generation / Autoregressive and latent generation
Improving the Diffusability of Autoencoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that modern autoencoders' latents carry excess high-frequency energy hurting diffusability, and fixes it by fine-tuning AEs with a scale-equivariance regularizer that matches decoder outputs on bilinearly downsampled latents and images.
Problem
Latent diffusion models built on modern autoencoders generate poorly because AE latent spectra contain excessive high-frequency components that disrupt the coarse-to-fine spectral autoregression of diffusion, a property the paper calls diffusability.
Approach
Enforce scale equivariance in the decoder by adding a loss that makes the decoder's reconstruction of a bilinearly downsampled latent match the downsampled RGB image (random 2x or 4x), which suppresses latent high frequencies without a KL term and without changing the encoder architecture or compute much.
Key results
- On ImageNet 256x256 without guidance, DiT-XL/2 on FluxAE+FT-SE reaches FID 9.85 vs 12.21 for vanilla FluxAE (about 19% lower in the abstract's framing, relative to vanilla), and FDD 235.8 vs 282.8.
- On Kinetics-700, DiT-B/2 on CogVideoX-AE with SE reduces FVD_10K from 650.40 to 252.26 (about 61%) and LTX-AE DiT-B/1 from 854.47 to 389.56, i.e. at least 44% reduction across video AEs.
- Reconstruction is largely preserved: FluxAE PSNR 30.24 vs 30.47 after SE fine-tuning, while FID rises from 0.183 to 0.550.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add scale equivariance when fine-tuning FluxAE | DiT-XL/2 with FluxAE · ImageNet-1K 256² | FID: 12.21 -> 9.85 | Helped |
| fine-tune FluxAE without scale equivariance | DiT-XL/2 with FluxAE · ImageNet-1K 256² | FID: 12.21 -> 10.62 | Helped |
| add scale equivariance to FluxAE fine-tuning | DiT-XL/2 with FluxAE · ImageNet-1K 256² | FID: 10.62 -> 9.85 | Helped |
| fine-tune CMS-AE-I without scale equivariance | DiT-B/1 with CMS-AE-I · ImageNet-1K 256² | FID: 11.69 -> 13.59 | Hurt |
| add scale equivariance to CMS-AE-I fine-tuning | DiT-B/1 with CMS-AE-I · ImageNet-1K 256² | FID: 11.69 -> 11.85 | Mixed |
| add scale equivariance when fine-tuning CogVideoX-AE | DiT-B/2 with CogVideoX-AE · Kinetics-700 17×256² | FVD10K: 650.40 -> 252.26 | Helped |
| add scale equivariance when fine-tuning LTX-AE | DiT-B/1 with LTX-AE · Kinetics-700 17×256² | FVD10K: 854.47 -> 389.56 | Helped |
| increase KL weight to β = 0.1 (ablation) | FluxAE fine-tuning with DiT-S/2 · ImageNet-1K | FDD5K: 992.05 -> 929.58 | Helped |
| increase KL weight to β = 0.1 (ablation) | FluxAE fine-tuning with DiT-L/2 · ImageNet-1K | FDD5K: 415.87 -> 472.74 | Hurt |
| use scale equivariance instead of vanilla FluxAE (ablation) | DiT-S/2 and DiT-L/2 with FluxAE · ImageNet-1K | FDD5K: 992.05 -> 924.28; 415.87 -> 369.15 | Helped |
| vary scale-equivariance loss weight α (ablation) | FluxAE fine-tuning with DiT-B/2 · ImageNet-1K | FID5k: 33.99; 33.86; 28.62; 26.84; 29.63; 33.22 | Mixed |
| replace downsampling regularization with explicit DCT high-frequency chopping (ablation) | FluxAE with DiT-S/2 · ImageNet-1K | FDD5K: 992.0 -> 912.4 | Helped |
Surprising findings
- Standard KL regularization can increase high-frequency latent components and hurt diffusability, and for larger DiT models it degrades results despite helping small ones.
- Temporal scale-equivariance regularization did not improve results, unlike the spatial version.
Limitations
- Regularization slightly worsens reconstruction FID for some autoencoders (FluxAE FID 0.183 to 0.550), and adversarial losses were omitted.
- Sensitive to hyperparameters such as regularization weight and KL term; training pipelines of the explored autoencoders were not released, so recipes had to be reconstructed.
- Only tested on four autoencoders at 256^2 resolution with DiT backbones; other parametrizations or architectures untested.
Experimental setup
Datasets: ImageNet-1K Kinetics-700 in-house image and video datasets COYO Panda-70M
Benchmarks: ImageNet-1K 256x256 Kinetics-700 17x256x256 ImageNet-1K 512x512
Metrics: FID FDD (DINOv2 Frechet distance) FVD_10K PSNR SSIM LPIPS
Baselines: FluxAE vanilla FluxAE + FT CMS-AE CogVideoX-AE LTX-AE SD-VAE-ft-MSE DiT-XL/2
Builds on: Latent diffusion models DiT rectified flow VAE KL regularization LPIPS
Closest papers by meaning
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.90)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.89)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
- Spectral Progressive Diffusion for Efficient Image and Video Generation (NeurIPS 2026 · similarity 0.89)
- Simpler Diffusion: 1.5 FID on ImageNet512 with Pixel-space Diffusion (CVPR 2025 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.