Research map / Image generation / Autoregressive and latent generation
What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prior-Aligned Autoencoder (PAE), a tokenizer that explicitly shapes the latent manifold via VFM-prior alignment (spatial structure, perturbation-based continuity, global semantics) to improve latent diffusion convergence and gFID on ImageNet 256.
Problem
Existing tokenizers for latent diffusion are designed for reconstruction or inherit VFM representations, leaving unclear what latent manifold structure is friendly for generative modeling.
Approach
Identifies three diffusion-friendly latent manifold properties (spatial structure coherence, local perceptual continuity, global semantic quality) and turns them into explicit tokenizer training objectives using refined frozen VFM priors, a detail-aware modulator, and cascaded perturbation consistency on a sphere-normalized latent.
Key results
- PAE (DINOv2) reaches 1.27 guided gFID at 80 epochs, comparable to RAE with up to 13x fewer epochs.
- PAE (DINOv2) achieves a new state-of-the-art 1.03 gFID at 800 epochs with guidance on ImageNet 256.
- PAE achieves rFID of 0.26, and 1.05 gFID with only 45 denoising steps.
Surprising findings
- Better reconstruction (rFID) alone does not guarantee better generation (gFID) across tokenizers.
Limitations
- Geometry metrics are empirical diagnostics rather than formally derived; causal link to diffusion difficulty is not established.
- Refinement of VFM targets gives only modest gains relative to prior alignment losses.
Experimental setup
Datasets: ImageNet COCO Panoptic Val 2017
Benchmarks: ImageNet 256x256
Metrics: gFID rFID IS Precision Recall SSC LPC GSQ eRank
Baselines: DDT REPA REPA-E REG SFD SVG VA-VAE VFM-VAE AlignTok RAE Send-VAE RPiAE
Builds on: REPA VA-VAE RAE LightningDiT LPIPS
Closest papers by meaning
- Isometric Representation Learning for Disentangled Latent Space of Diffusion Models (ICML 2024 · similarity 0.91)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.89)
- How Diffusion Models Learn to Factorize and Compose (NeurIPS 2024 · similarity 0.89)
- Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion (ICLR 2025 · similarity 0.89)
- Manifold Preserving Guided Diffusion (ICLR 2024 · similarity 0.89)
- Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.