Research map / Image generation / Autoregressive and latent generation
IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDEAL, a discrete representation autoencoder that fuses shallow and deep frozen SigLIP2 features before vector quantization and supervises the decoded tokens to reconstruct both, achieving strong rFID and AR generation gFID on ImageNet.
Problem
Discrete tokenizers built on frozen vision foundation models (VFMs) preserve semantics from deep layers but lose fine-grained visual detail, hurting reconstruction and autoregressive image generation after quantization.
Approach
Fuse shallow (appearance-rich) and deep (semantic) frozen VFM features via cross-attention before vector quantization, then decode tokens back into both feature spaces with alignment losses, so one discrete code retains both detail and semantics.
Key results
- Tokenizer achieves rFID 0.61 and rIS 230.4 on ImageNet reconstruction with 100% codebook usage, outperforming prior tokenizers in the table.
- Decoded interface features reach 80.89% zero-shot ImageNet Top-1 versus 83.23% for SigLIP2.
- AR generation with IDEAL-3B (3.1B params) reaches gFID 1.89 on ImageNet 256x256 with CFG, reported as new state of the art among AR models.
Surprising findings
- Removing fusion keeps rIS nearly unchanged (231.1 vs 230.4) while rFID worsens, and the base IDEAL-B has lower gIS than VFMTok-B despite better gFID.
- Under a controlled 256 setting, IDEAL-B's gIS (181.9) is well below VFMTok-B (247.6) even though its gFID is better (3.43 vs 3.61).
Limitations
- Tokenizer trained only on ImageNet, so reconstruction degrades on faces, text, and long-tail content.
- Semantic preservation evaluated only via ImageNet zero-shot classification, not broader semantic benchmarks.
- Generation artifacts remain in hands, faces, and fine structures; lower gIS than VFMTok at similar scale.
Experimental setup
Datasets: ImageNet-1K LLaVA SFT data
Benchmarks: ImageNet-1K RealWorldQA ChartQA OKVQA InfoVQA SEED MME
Metrics: rFID rIS gFID gIS sFID Precision Recall zero-shot Top-1/Top-5 linear probing Top-1
Baselines: VFMTok LlamaGen VQRAE DINO-Tok VAR TiTok ImageFolder ViT-VQGAN MaskGiT DiT SiT FasterDiT
Builds on: RAE VFMTok VQGAN LlamaGen VAR
Closest papers by meaning
- Improved Baselines with Representation Autoencoders (NeurIPS 2026 · similarity 0.88)
- One Unified Representation: Resolving the Appearance-Semantics Dilemma via Structural Regularization (NeurIPS 2026 · similarity 0.88)
- dRAE: Representation Autoencoder with Hyper-Spherical Codes (NeurIPS 2026 · similarity 0.87)
- DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders (NeurIPS 2026 · similarity 0.86)
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.86)
- Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.