Research map / Image generation / Autoregressive and latent generation

Improved Baselines with Representation Autoencoders

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Hierarchical VQ-VAE generation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Improves Representation Autoencoders (RAE) by summing the last K encoder layers, showing RAE and REPA are complementary, and turning the REPA head into a free x-prediction guidance signal, reaching gFID 1.06 in 80 epochs on ImageNet-256.

Problem

Representation Autoencoders (RAE) built on frozen pretrained vision encoders have weak reconstruction, are incompatible with classifier-free guidance, need a second model for AutoGuidance, and converge slowly; prior work also assumes RAE makes REPA redundant.

Approach

Define the encoder representation as the sum of the last K layers (training-free), show RAE and REPA work through complementary mechanisms so the same encoder can serve as both latent and REPA target, and reparameterize the REPA head as x-prediction so it can serve as a weak model for internal guidance within a single forward pass.

Key results

• RAEv2 with REPA guidance reaches gFID 1.06 on ImageNet-256 in 80 epochs, versus RAE's 1.13 at 800 epochs.
• RAEv2 reduces epochs to reach unguided gFID<=2 from 177 (RAE) to 35 epochs; FD_r^6 of 2.17 at 80 epochs vs 3.26 for RAE at 800 epochs.
• On RECON world-model rollouts up to 16s, RAEv2-NWM achieves FVD 105.61 vs 312.01 for RAE and 200.97 for NWM.

Surprising findings

• REPA combined with RAE consistently improves generation across all 27 tested encoders rather than being redundant.
• Stronger encoders that were worse under the original RAE recipe (e.g., DINOv3) perform better under RAEv2, reversing the encoder ranking.

Limitations

• Text-to-image results use a different encoder (SigLIP2-B) and short schedules, so gains are not fully isolated.
• Reconstruction gains at high K hurt generation without guidance (K=23 gFID 3.02 vs K=7 1.65), so K must be tuned for the trade-off.

Experimental setup

Datasets: ImageNet-1K JourneyDB BLIP3o BLIP3o-60k RECON

Benchmarks: ImageNet-256 GenEval DPG-Bench GenAI-Bench RECON

Metrics: gFID IS rFID PSNR FD_r^6 epochs to gFID<=2 (eFID) GenEval DPG FVD FID

Baselines: RAE REPA-E RAE-XL SiT-XL/2 DDT-XL LightningDiT REG Flux-VAE CFG AutoGuidance NWM DIAMOND

Builds on: RAE REPA DDT AutoGuidance iREPA

Closest papers by meaning

• Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think (NeurIPS 2025 · similarity 0.88)
• IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder (NeurIPS 2026 · similarity 0.88)
• REPA Works Until It Doesn’t: Early-Stopped, Holistic Alignment Supercharges Diffusion Training (NeurIPS 2025 · similarity 0.87)
• Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.87)
• DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders (NeurIPS 2026 · similarity 0.87)
• DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence (NeurIPS 2026 · similarity 0.87)