Research map / Image generation / Autoregressive and latent generation
Epsilon-VAE: Denoising as Visual Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ε-VAE, which replaces the VAE decoder with a conditional rectified-flow diffusion decoder trained with score/velocity, LPIPS and trajectory-matching adversarial losses, yielding better rFID and downstream DiT generation FID with 1-3 sampling steps.
Problem
Standard autoencoder tokenizers reconstruct images in a single deterministic decoding step, limiting reconstruction fidelity and the quality of latents for downstream latent diffusion generation, especially at high compression.
Approach
Replace the deterministic decoder with a conditional diffusion (rectified flow) decoder that iteratively denoises noise into the image, conditioned on encoder latents by channel concatenation, and adapt LPIPS and GAN losses to a start-to-end denoising trajectory matching objective with logit-normal time sampling, input scaling, and reversed-log inference time spacing.
Key results
- On ImageNet 256x256 at 8x8 downsampling, ε-VAE-SD (B) one-step rFID 0.57 vs SD-VAE 0.74, and (M) 0.51 with PSNR 26.45 vs 25.68.
- Class-conditional generation with DiT-XL/2 at 8x8 downsampling: ε-VAE-SD (H) FID 8.85 vs SD-VAE 11.63; at 16x16 ε-VAE-SD (H) FID 9.72 vs SD-VAE 14.59 with ~1.0x throughput of SD-VAE at 32x32 compression reaching 2.3x throughput claim.
- Ablation on ε-VAE-lite: rFID drops from 28.22 (DDPM baseline) to 6.24 with reversed-log time spacing and 3 NFE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single-step VAE decoding with conditional diffusion decoding | ImageNet autoencoding and latent generation · ImageNet 256×256, downsample 8×8 | rFID: 0.74 -> 0.51 | Helped |
| Replace standard diffusion parameterization with rectified flow (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 28.22 -> 24.11 | Helped |
| Replace uniform with logit-normal training time sampling (ablation) | ε-VAE-lite rectified-flow decoder · ImageNet 128×128 | rFID: 24.11 -> 23.44 | Helped |
| Replace DDPM UNet with ADM UNet (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 23.44 -> 22.04 | Helped |
| Add perceptual matching on estimated and real images (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 22.04 -> 11.76 | Helped |
| Add adversarial start-to-end denoising trajectory matching (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 11.76 -> 8.24 | Helped |
| Scale noisy states by γ=0.6 (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 8.24 -> 7.08 | Helped |
| Replace uniform with reversed logarithm inference time spacing (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 7.08 -> 6.24 | Helped |
| Replace channel concatenation with AdaGN conditioning (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 22.04 -> 22.01 | No clear effect |
| Replace start-to-end trajectory matching with adjacent-step matching (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID: 8.24 -> 10.53 | Hurt |
| Replace start-to-end trajectory matching with uncoupled image-distribution matching (ablation) | ε-VAE-lite diffusion decoder · ImageNet 128×128 | rFID | Hurt |
| Use adaptive weighting for the adversarial-loss coefficient (ablation) | ε-VAE discriminator training · ImageNet autoencoding | reconstruction quality | No clear effect |
Surprising findings
- Adaptive loss weighting for the GAN term gave no benefit, so a fixed lambda_adv = 0.5 was used.
- Matching the distribution of predicted x0 to x0 without coupling on x_t led to unstable training and no reasonable results.
Limitations
- UNet-based diffusion decoder has memory and throughput overhead at high resolutions.
- Applying LPIPS and GAN losses to one-step estimates may introduce objective bias.
Experimental setup
Datasets: ImageNet COCO-2017
Benchmarks: ImageNet 256x256 reconstruction ImageNet 128x128 reconstruction ImageNet 512x512 reconstruction COCO-2017 reconstruction ImageNet class-conditional generation 256x256
Metrics: rFID FID PSNR SSIM IS precision recall throughput
Baselines: SD-VAE LiteVAE VQGAN LlamaGen SDXL-VAE Asym-VAE Omni-VAE MaskGIT ViT-VQGAN
Builds on: VQGAN rectified flow ADM DiT LPIPS
Closest papers by meaning
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.90)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.90)
- Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation (NeurIPS 2025 · similarity 0.90)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.90)
- SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer (CVPR 2025 · similarity 0.89)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.