Research map / Image generation / Autoregressive and latent generation
ImageFolder: Autoregressive Image Generation with Folded Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ImageFolder, a semantic image tokenizer with dual-branch product quantization and quantizer dropout whose spatially aligned tokens are folded in pairs during next-scale AR generation, halving the sequence length (680→286) while improving ImageNet 256 generation.
Problem
Longer image token sequences improve reconstruction but hurt autoregressive generation quality and cost, creating a reconstruction–generation trade-off in visual tokenizers for AR models.
Approach
Use product quantization with two branches (one semantically regularized via CLIP-style alignment to DINOv2, one capturing pixel details) to produce spatially aligned token pairs with low mutual dependency, then predict both tokens from a single logit ("folding") so the AR sequence is halved; quantizer dropout further smooths next-scale prediction.
Key results
- ImageFolder achieves gFID 2.60 and rFID 0.80 on ImageNet 256x256 with a 286-token sequence, versus VAR's 680 tokens (gFID 3.30, rFID 0.90).
- Ablation shows dual-branch (one semantic, one detail) product quantization with parallel decoding reduces gFID from 5.21 (semantic on all branches) to 3.53 at 286 tokens, before adding the DINO discriminator.
- Inference latency with batch size 1 on A100 is 0.130s for ImageFolder vs 0.134s for VAR and 8.851s for LlamaGen, with about 1/4 of VAR's FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add quantizer dropout (p=0.1) to multi-scale residual quantization (ablation) | ImageFolder tokenizer · ImageNet 256x256 validation | gFID: 7.52 -> 6.03 | Helped |
| Reduce quantized token size to K=11 (ablation) | ImageFolder tokenizer · ImageNet 256x256 validation | rFID: 1.71 -> 3.24 | Hurt |
| Add product quantization and parallel decoding (ablation) | ImageFolder tokenizer and VAR · ImageNet 256x256 validation | gFID: 6.03 -> 5.96 | Helped |
| Apply semantic regularization to all branches (ablation) | ImageFolder tokenizer · ImageNet 256x256 validation | gFID: 5.96 -> 5.21 | Helped |
| Restrict semantic regularization to one branch (ablation) | ImageFolder tokenizer · ImageNet 256x256 validation | gFID: 5.21 -> 3.53 | Helped |
| Add the DINO discriminator (ablation) | ImageFolder tokenizer · ImageNet 256x256 validation | gFID: 3.53 -> 2.60 | Helped |
| Set quantizer dropout start N_start to 1 instead of 3 (ablation) | ImageFolder tokenizer · ImageNet validation | rFID: 1.57 -> $>$20 | Hurt |
| Set quantizer dropout start N_start to 5 instead of 3 (ablation) | ImageFolder tokenizer · ImageNet validation | rFID: 1.57 -> 1.69 | Hurt |
| Set quantizer dropout ratio p to 0 instead of 0.1 (ablation) | ImageFolder tokenizer · ImageNet validation | rFID: 1.57 -> 1.73 | Hurt |
| Increase quantizer dropout ratio p to 0.5 (ablation) | ImageFolder tokenizer · ImageNet validation | rFID: 1.57 -> 1.94 | Hurt |
| Increase product-quantization branches from 2 to 4 (ablation) | ImageFolder tokenizer · ImageNet validation | gFID: 3.53 -> 5.24 | Hurt |
| Remove semantic loss (lambda_clip=0) (ablation) | ImageFolder tokenizer · ImageNet validation | rFID: 1.57 -> 1.97 | Hurt |
Surprising findings
- Semantic regularization applied to all product branches degrades generation relative to applying it to a single branch, and DINOv2 features alone as a tokenizer yield rFID above 5 under linear tuning.
- Using a 4-branch product quantizer improves rFID to 0.97 but worsens gFID (5.24 reported for 4 branches), showing reconstruction and generation trade off across branch count.
Limitations
- Tokenizer training recipe is not advanced (fewer iterations, weaker discriminator); the VAR comparison uses a tokenizer trained with stronger recipe on OpenImages, so results are not directly comparable.
- Semantic regularization on all branches or using more branches can hurt generation, so the design is sensitive to branch configuration.
- Evaluation limited to ImageNet 256x256 class-conditional generation.
Experimental setup
Datasets: ImageNet OpenImages
Benchmarks: ImageNet 256x256 class-conditional generation
Metrics: rFID gFID IS Precision Recall linear probing accuracy inference latency
Baselines: VAR LlamaGen-L MaskGIT MAR-B DiT-L/2 LDM-4 MAGVIT-v2 TiTok-S-128 RQ-Transformer VQGAN
Builds on: VAR VQGAN LlamaGen DINOv2 TiTok
Closest papers by meaning
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.91)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.90)
- Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (NeurIPS 2024 · similarity 0.89)
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.89)
- An Image is Worth 32 Tokens for Reconstruction and Generation (NeurIPS 2024 · similarity 0.89)
- UniTok: a Unified Tokenizer for Visual Generation and Understanding (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.