Research map / Image generation / Autoregressive and latent generation
Adaptive Length Image Tokenization via Recurrent Allocation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALIT, a recurrent encoder-decoder that distills frozen VQGAN 2D tokens into a variable number of 1D latent tokens (32 to 256) by adding new latent tokens each iteration, trained with reconstruction and GAN losses.
Problem
Vision tokenizers assign fixed-length representations to images regardless of information content, tying token count to the patch grid and preventing adaptive per-image representational capacity.
Approach
Recurrently roll out a shared 2D-to-1D distillation encoder-decoder over VQGAN tokens, adding new learnable latent tokens at each iteration and residually updating existing ones, so token count grows adaptively (32 to 256) with reconstruction as the self-supervised objective.
Key results
- Ours-S FID on ImageNet100 drops from 22.57 at 32 tokens to 8.25 at 256 tokens, versus VQGAN 7.04 and TiTok-L-32 11.60 at 32 tokens.
- Linear probe top-1 on ImageNet-1K: Ours-S-32 49.9% vs TiTok-S-32 48.0%; Ours-SemiLarge 59.5% vs TiTok-L-32 60.0%.
- Emergent segmentation on ImageNet-S reaches 71.8 mIoU with per-image adaptive thresholding and 57.8 mIoU at fixed 40% attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the token count from 32 to 256 | ALIT-S · ImageNet-100 | FID: 22.57 -> 8.25 | Helped |
| Increase the token count from 32 / 64 to 256 | ALIT-S · COCO | FID: 22.28 -> 9.72 | Helped |
| Increase the token count from 32 / 64 to 256 | ALIT-S · Wikipedia (WIT) | FID: 61.77 -> 38.45 | Helped |
| Use ALIT-S-32 instead of Titok-S-32 | ImageNet-1K linear probing · ImageNet-1K | Top-1 accuracy: 48.0\% -> 49.9\% | Helped |
| Use ALIT-SemiLarge instead of Titok-L-32 | ImageNet-1K linear probing · ImageNet-1K | Top-1 accuracy: 60.0\% -> 59.5\% | Hurt |
| Apply recurrent encoder processing for 2–3 iterations instead of one (ablation) | ALIT-L · ImageNet-1K | Top-1 accuracy: 55.6\% -> 59.5\% | Helped |
| Increase recurrent processing from iteration 1 to iteration 8 (ablation) | ALIT token attention maps · ImageNet-S segmentation alignment, 40% attention threshold | mIOU: 56.7 -> \textbf{57.8} | Helped |
| Use continuous 1D latents instead of quantized 1D latents (ablation) | ALIT with VQGAN 2D tokenizer · ImageNet-100 | FID: 22.57 -> 21.77; 16.17 -> 10.39; 13.30 -> 8.02; 11.69 -> 7.38; 10.22 -> 7.30; 9.30 -> 7.44; 8.55 -> 7.36; 8.25 -> 7.14 | Helped |
| Use continuous 1D latents instead of quantized 1D latents (ablation) | ALIT with VAE 2D tokenizer · ImageNet-100 | FID: 27.90 -> 22.14; 19.62 -> 14.15; 16.07 -> 10.88; 13.95 -> 9.00; 12.50 -> 7.68; 11.16 -> 6.76; 12.62 -> 6.22; 9.95 -> 5.66 | Helped |
| Increase model depth from small (8) to base (12) (ablation) | ALIT Stage1, embed dim 512 to 768 · ImageNet-100, 32 tokens | FID: 71.53 -> 49.88 | Helped |
| Increase model depth from base (12) to semi-large (16) (ablation) | ALIT Stage1 · ImageNet-100, 32 tokens | FID: 49.88 -> 43.97 | Helped |
| Fine-tune in Stage2 instead of using Stage1 only (ablation) | ALIT small (8), embed dim 1024 · ImageNet-100 | FID: 53.47 -> 22.57; 35.66 -> 16.17; 27.51 -> 13.30; 21.18 -> 11.69; 16.66 -> 10.22; 12.94 -> 9.30; 9.66 -> 8.55; 7.29 -> 8.2 | Mixed |
Surprising findings
- Weaker downstream models tolerate fewer tokens with smaller performance drops, suggesting token needs depend on the downstream model's strength.
- Continuous 1D latents without quantization give better reconstruction than quantized latents, at the cost of more bits per image.
Limitations
- FID at low token counts (32-64) lags behind fixed-length baselines like TiTok and VQGAN.
- Uses a frozen-then-finetuned pretrained VQGAN as base tokenizer, and FID experiments train on ImageNet-100 only due to compute.
- Object discovery is evaluated only through emergent attention-to-segmentation alignment, not optimized or benchmarked against dedicated segmentation methods.
Experimental setup
Datasets: ImageNet-100 ImageNet-1K COCO Wikipedia Image-Text (WIT) PeopleART SAVOIAS NYUV2 ImageNet-S
Benchmarks: ImageNet-100 reconstruction FID COCO reconstruction FID WIT reconstruction FID ImageNet-1K linear probing ImageNet-S segmentation mIoU
Metrics: FID L1 reconstruction loss top-1 accuracy mIoU depth error
Baselines: TiTok-L-32 TiTok-B-64 TiTok-S-128 VQGAN
Builds on: VQGAN Perceiver TiTok Matryoshka Representation Learning Universal Transformers
Closest papers by meaning
- FlexTok: Resampling Images into 1D Token Sequences of Flexible Length (ICML 2025 · similarity 0.90)
- Single-pass Adaptive Image Tokenization for Minimum Program Search (NeurIPS 2025 · similarity 0.90)
- ElasticTok: Adaptive Tokenization for Image and Video (ICLR 2025 · similarity 0.88)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.88)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.88)
- Subobject-level Image Tokenization (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.