Research map / Image generation / Autoregressive and latent generation
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VA-VAE, a continuous VAE whose latent space is aligned to frozen vision foundation models (DINOv2, MAE) via a margin-based VF loss, plus LightningDiT, an optimized DiT baseline, to resolve the reconstruction-generation dilemma in latent diffusion.
Problem
Higher-dimensional visual tokenizers improve reconstruction but substantially slow diffusion transformer convergence and degrade generation quality, creating an optimization dilemma in latent diffusion models.
Approach
Align the high-dimensional VAE latent space with pre-trained vision foundation model features using a marginal cosine similarity loss and a marginal distance-matrix similarity loss (VF loss) with adaptive gradient-based weighting, so the tokenizer keeps reconstruction quality while producing a more generation-friendly latent space; pair it with an optimized DiT (LightningDiT).
Key results
- Integrated system reaches FID 1.35 on ImageNet 256x256 class-conditional generation (800 epochs, with CFG), with rFID 0.28 for VA-VAE.
- LightningDiT with VA-VAE reaches FID 2.11 in 64 epochs, about 21.8x faster convergence than original DiT (1400 epochs).
- VF loss improves gFID on f16d64 tokenizer from 45.81 to 29.86 (DiT-B, 80 epochs) with minimal reconstruction loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align the f16d32 tokenizer with DINOv2 using VF loss | LDM with DiT-B · ImageNet 256×256 | gFID-10k: 31.90 -> 19.93 | Helped |
| Align the f16d64 tokenizer with DINOv2 using VF loss | LDM with DiT-B · ImageNet 256×256 | gFID-10k: 45.81 -> 29.86 | Helped |
| Increase tokenizer dimension from f16d16 to f16d64 | LDM with DiT-B · ImageNet 256×256 | gFID-10k: 21.20 -> 45.81 | Hurt |
| Increase tokenizer dimension from f16d16 to f16d64 | LDM · ImageNet 256×256 | rFID-50k: 0.43 -> 0.18 | Helped |
| Use DINOv2 rather than no VF loss (ablation) | VA-VAE with LightningDiT-B · ImageNet 256×256 | gFID: 29.35 -> 24.77 | Helped |
| Use MAE rather than no VF loss (ablation) | VA-VAE with LightningDiT-B · ImageNet 256×256 | gFID: 29.35 -> 26.44 | Helped |
| Use SAM rather than no VF loss (ablation) | VA-VAE with LightningDiT-B · ImageNet 256×256 | gFID: 29.35 -> 29.13 | No clear effect |
| Use CLIP rather than no VF loss (ablation) | VA-VAE with LightningDiT-B · ImageNet 256×256 | gFID: 29.35 -> 27.46 | Helped |
| Add marginal cosine similarity loss (ablation) | DINOv2-aligned f16d32 tokenizer · ImageNet 256×256 | gFID: 29.35 -> 28.92 | Helped |
| Add marginal distance matrix similarity loss (ablation) | DINOv2-aligned f16d32 tokenizer · ImageNet 256×256 | gFID: 29.35 -> 28.46 | Helped |
| Use both marginal cosine and marginal distance matrix similarity losses (ablation) | DINOv2-aligned f16d32 tokenizer · ImageNet 256×256 | gFID: 29.35 -> 24.77 | Helped |
| Apply VF loss to high-dimensional tokenizers | LightningDiT-B · ImageNet 256×256 | convergence speedup: 1.8 and 2.7 | Helped |
Surprising findings
- Naively initializing VAE encoders with pretrained vision foundation models was reported as ineffective, requiring the joint reconstruction and alignment loss design.
Limitations
- Gains are evaluated mainly on class-conditional ImageNet 256x256, not text-to-image settings.
- Optimal margin settings differ across vision foundation models and require manual tuning (m2 set differently for DINOv2).
Experimental setup
Datasets: ImageNet
Benchmarks: ImageNet 256x256
Metrics: FID gFID rFID PSNR LPIPS SSIM IS sFID precision recall
Baselines: DiT-XL/2 SiT-XL/2 MDT-XL/2 REPA-XL/2 MAR LlamaGen VAR MaskDiT MDTv2 FasterDiT
Builds on: latent diffusion model DiT VQGAN REPA rectified flow
Closest papers by meaning
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.92)
- Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (ICLR 2025 · similarity 0.92)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.90)
- Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.90)
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.90)
- LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.