Research map / Architectures and efficiency / RNNs and normalization
Single-pass Adaptive Image Tokenization for Minimum Program Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KARL, a single-pass adaptive image tokenizer that predicts how many 1D tokens an image needs to reach a target reconstruction loss, trained with a loss-conditioned two-run (upside-down RL-style) scheme and framed as an approximation of Kolmogorov Complexity.
Problem
Existing adaptive image tokenizers pick the number of tokens per image only through test-time iterative search over multiple encoder-decoder passes, which is costly for real-time uses like VLMs and video models.
Approach
Condition the encoder on a target reconstruction loss obtained from a first near-lossless pass, and train it in a second pass with extra tokens to halt surplus tokens via a BCE halting loss, so the minimal token count is predicted in one forward pass; the token count is framed as an approximate Kolmogorov Complexity.
Key results
- Under the same token count, KARL achieves L1 x10 of 0.70 at 256 tokens vs ALIT's 0.76 and LPIPS of 0.14 vs 0.16 on ImageNet100.
- With variable token allocation at threshold 0.05, KARL uses one encoder+decoder pass (1+1) while ALIT needs about 7.01x2 runs, with comparable LPIPS (0.15 vs 0.17).
- KARL has higher FID than ALIT and One-D-Piece in the variable regime (10.3 at 0.05 vs 8.3 for One-D-Piece), so it trails on FID.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KARL instead of ALIT for fixed-count reconstruction | ImageNet100 reconstruction · ImageNet100, fixed 128 tokens (IN100-trained) | LPIPS: 0.20 -> 0.19 | Helped |
| Use KARL instead of ALIT for fixed-count reconstruction | ImageNet100 reconstruction · ImageNet100, fixed 128 tokens (IN100-trained) | Dreamsim: 0.14 -> 0.16 | Hurt |
| Use KARL instead of ALIT for variable-token reconstruction | ImageNet100 adaptive tokenization · ImageNet100, ε=0.05 (IN100-trained) | LPIPS: 0.17 -> 0.15 | Helped |
| Use KARL instead of ALIT for variable-token reconstruction | ImageNet100 adaptive tokenization · ImageNet100, ε=0.05 (IN100-trained) | Encoder + decoder runs: 7.01$\times$ 2 -> 1 $+$ 1 | Helped |
| Use KARL instead of One-D-Piece for variable-token reconstruction | ImageNet100 adaptive tokenization · ImageNet100, ε=0.05 (IN100-trained) | LPIPS: 0.19 -> 0.16 | Helped |
| Use KARL instead of One-D-Piece | ImageNet100 reconstruction · ImageNet100, variable tokens, ε=0.05 | FID: 8.3 -> 10.3 | Hurt |
| Replace quantized 1D tokens with continuous 1D tokens (ablation) | KARL · Image reconstruction, fixed and variable token allocation | FID | Helped |
| Use continuous rather than quantized 1D tokens (ablation) | KARL · Adaptive token allocation | Average token count | Helped |
| Change the 2D tokenizer from VAE to VQGAN while keeping 1D tokens quantized (ablation) | KARL · Adaptive image reconstruction | Minimum token count | No clear effect |
| Pair continuous 1D tokens with a discrete 2D tokenizer (ablation) | KARL · Adaptive image reconstruction | Minimum token count | Hurt |
| Use a small encoder with a large decoder rather than a large encoder with a large decoder (ablation) | KARL · Image reconstruction, variable token allocation | Reconstruction performance | Helped |
| Use a small encoder rather than a large encoder (ablation) | KARL · Image reconstruction, fixed token allocation | Reconstruction performance | No clear effect |
Surprising findings
- A small encoder with a large decoder performs best, and variable token allocation reveals this gap more clearly than fixed token counts.
- Continuous 1D tokens outperform quantized 1D tokens even under adaptive per-image allocation.
Limitations
- Trails prior adaptive tokenizers on FID, with the authors attributing this to GAN-based training and blurry One-D-Piece outputs.
- Token count is only an approximation of Kolmogorov Complexity; no minimality guarantees, and the AIT connection is conceptual rather than formal.
- Experiments are mostly on the ImageNet100 subset, with a limited set of datasets.
Experimental setup
Datasets: ImageNet-1K ImageNet100 Savoias
Benchmarks: ImageNet100
Metrics: L1 loss LPIPS SSIM DreamSim FID PSNR tokens used encoder+decoder runs
Baselines: ALIT One-D-Piece ElasticTok FlexTok
Builds on: ALIT One-D-Piece VQGAN Perceiver Upside-Down Reinforcement Learning
Closest papers by meaning
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.90)
- TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
- Subobject-level Image Tokenization (ICML 2025 · similarity 0.87)
- End-to-End Vision Tokenizer Tuning (NeurIPS 2025 · similarity 0.87)
- CAT: Content-Adaptive Image Tokenization (NeurIPS 2025 · similarity 0.87)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.