Research map / Image generation / Autoregressive and latent generation
An Image is Worth 32 Tokens for Reconstruction and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TiTok, a ViT-based 1D tokenizer that encodes a 256x256 image into 32 discrete latent tokens, trained in two stages with MaskGIT-VQGAN proxy codes, and used with MaskGIT for class-conditional ImageNet generation.
Problem
Standard VQ image tokenizers produce 2D latent grids tied to image patches, which limits exploitation of spatial redundancy and yields many tokens (256 to 1024), making generation slow and costly.
Approach
Replace the 2D latent grid with a 1D sequence of learnable latent tokens appended to patch embeddings in a ViT encoder, decoded by a ViT decoder fed with mask tokens; train in two stages, first regressing proxy codes from an off-the-shelf MaskGIT-VQGAN and then fine-tuning the decoder to pixels.
Key results
- TiTok-L-32 reaches 2.77 gFID on ImageNet 256x256 with 32 tokens versus 6.18 for MaskGIT with the same generator, and 1.97 gFID for TiTok-S-128 with 64 steps.
- TiTok-L-64 at 512x512 reaches 2.74 gFID vs DiT-XL/2 3.04 while running 410x faster in sampling throughput (41.0 vs 0.1 samples/s).
- TiTok-L-32 achieves rFID 2.21, comparable to MaskGIT-VQGAN (2.28) with 8x fewer latent tokens; two-stage training gives TiTok-B-64 rFID 1.70.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MaskGIT-VQGAN tokenizer with TiTok-L-32 while keeping MaskGIT-ViT and sampling steps the same | MaskGIT-ViT · ImageNet-1K 256×256 | gFID: 6.18 to 2.77 | Helped |
| use TiTok-S-128 instead of DiT-XL/2 | ImageNet generation · ImageNet-1K 256×256 | gFID: 2.27 vs. 1.97 | Helped |
| use TiTok-L-64 instead of DiT-XL/2 | ImageNet generation · ImageNet-1K 512×512 | gFID: 3.04 vs. 2.74 | Helped |
| use TiTok-B-128 instead of DiT-XL/2 | ImageNet generation · ImageNet-1K 512×512 | gFID: 3.04 vs. 2.13 | Helped |
| increase the codebook size (ablation) | TiTok-L-32 · ImageNet-1K reconstruction | rFID: 6.59 -> 5.85 | Helped |
| increase training duration from 100 to 200 epochs (ablation) | TiTok-L-32 · ImageNet-1K reconstruction | rFID: 5.85 -> 5.48 | Helped |
| fine-tune the decoder (ablation) | TiTok-L-32 · ImageNet-1K reconstruction | rFID: 5.48 -> 2.21 | Helped |
| use arccos rather than cosine masking schedule (ablation) | TiTok-L-32 + MaskGIT · ImageNet-1K generation | gFID: 5.17 -> 4.94 | Helped |
| use linear rather than cosine masking schedule (ablation) | TiTok-L-32 + MaskGIT · ImageNet-1K generation | gFID: 5.17 -> 4.95 | Helped |
| use square-root rather than cosine masking schedule (ablation) | TiTok-L-32 + MaskGIT · ImageNet-1K generation | gFID: 5.17 -> 5.63 | Hurt |
| replace 1D latent tokens with a 2D image-patch representation (ablation) | TiTok-B-64 · ImageNet-1K reconstruction | rFID: 5.15 -> 15.58 | Hurt |
| use proxy codes and two-stage training instead of Taming-VQGAN single-stage training (ablation) | TiTok-B-64 · ImageNet-1K reconstruction | rFID: 5.15 -> 1.70 | Helped |
Surprising findings
- A 2D variant of the same architecture performs much worse than the 1D tokenizer at compact latent sizes (15.58 vs 5.15 rFID for TiTok-B-64 under the Taming-VQGAN setting).
- Masking schedules preferred by MaskGIT (cosine) are not the best for TiTok; arccos or linear work better.
Limitations
- Validated only with vanilla VQ tokenizer and MaskGIT generator; generalization to VAEs, diffusion, or video is untested.
- Classifier-free guidance benefits 1D compact tokens less than larger token sets, leaving potential untapped.
- Only class-conditional ImageNet generation evaluated; single-stage training underperforms MaskGIT-VQGAN without the proxy-code recipe.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K 256x256 ImageNet-1K 512x512
Metrics: gFID rFID IS linear probing accuracy throughput (samples/s)
Baselines: MaskGIT-VQGAN VQGAN (Taming-Transformer) DiT-XL/2 LDM-4 MAGVIT-v2 RQ-VAE ViT-VQGAN UViT
Builds on: VQGAN MaskGIT ViT MAE
Closest papers by meaning
- Language-Guided Image Tokenization for Generation (CVPR 2025 · similarity 0.90)
- Image Understanding Makes for A Good Tokenizer for Image Generation (NeurIPS 2024 · similarity 0.90)
- When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization (NeurIPS 2025 · similarity 0.90)
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.90)
- Highly Compressed Tokenizer Can Generate Without Training (ICML 2025 · similarity 0.89)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.