Research map / Vision-language and multimodal / Multimodal LLMs
ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ALTo, an adaptive-length mask tokenizer with a learned token length predictor and differentiable chunking, and ALToLLM, an MLLM that generates 1-32 mask tokens autonomously, refined with GRPO for quality-efficiency trade-offs.
Problem
Mask tokenizers for MLLM segmentation use fixed-length token sequences regardless of object complexity, wasting computation on simple masks; prior adaptive-length tokenizers rely on heuristic reconstruction-based rules that cannot be used for generation.
Approach
A token length predictor on top of a 1D mask tokenizer, trained with a soft differentiable token chunking (straight-through on cumulative stop probabilities) plus a linear length penalty, lets the model itself choose how many mask tokens to emit; an MLLM is then trained with SFT and GRPO using an IoU-plus-length-penalty reward.
Key results
- ALToLLM-8B (AL) reaches 75.4 cIoU on gRefCOCO val using 17.5 mask tokens on average, versus 74.8 cIoU at fixed 32 tokens for the FL variant.
- ALToLLM-8B (AL) achieves 85.8 / 86.6 / 84.7 cIoU on RefCOCO val / testA / testB, reported as state-of-the-art in the paper's comparison table.
- Adaptive-length models save more than 10 tokens at the same IoU level on Multi-Target-SA1B and gRefCOCO, and the advantage persists zero-shot on A-150.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive-length tokens instead of 32 fixed-length tokens | ALToLLM-8B on gRefCOCO val · gRefCOCO val | cIoU: 74.8 -> 75.4 | Helped |
| use adaptive-length tokens instead of 32 fixed-length tokens | ALToLLM-8B on gRefCOCO val · gRefCOCO val | average token length: 32 -> 17.5 | Helped |
| use adaptive-length tokens instead of 32 fixed-length tokens | ALToLLM-8B on gRefCOCO val · gRefCOCO val | generation time per sample: 1.079 -> 0.710 | Helped |
| use adaptive-length SFT | ALToLLM-8B on RefCOCO · RefCOCO val | cIoU: 84.9 -> 85.8 | Helped |
| use adaptive-length SFT | ALToLLM-8B on RefCOCO+ · RefCOCO+ testB | cIoU: 77.9 -> 77.0 | Hurt |
| remove pixel encoder (ablation) | ALTo reconstruction model on Multi-Target-SA1B · Multi-Target-SA1B val | gIoU: 94.4 -> 91.9 | Hurt |
| add Multi-Target-SA1B pretraining (ablation) | ALToLLM on gRefCOCO · gRefCOCO val | gIoU: 72.1 -> 75.6 | Helped |
| add pixel encoder (ablation) | ALToLLM on gRefCOCO · gRefCOCO val | gIoU: 75.6 -> 77.6 | Helped |
| use adaptive-length SFT (ablation) | ALToLLM on gRefCOCO · gRefCOCO val | average length: 32.0 -> 17.5 | Helped |
| add GRPO (ablation) | ALToLLM on gRefCOCO · gRefCOCO val | gIoU: 78.0 -> 78.1 | Helped |
| add GRPO (ablation) | ALToLLM on gRefCOCO · gRefCOCO val | average length: 17.5 -> 15.7 | Helped |
Surprising findings
- Adaptive-length SFT slightly improves mask quality over fixed 32-token SFT while roughly halving the token count.
- GRPO contributes only marginally to absolute metrics (gRefCOCO val cIoU 75.4 to 75.5) while reducing average length from 17.5 to 15.7.
Limitations
- Multi-stage training pipeline (pretrain tokenizer, fine-tune length predictor, SFT, GRPO) adds engineering complexity.
- Experiments focus on mask generation; extension to RGB image tokenization is left to future work.
- Length penalty must be tuned per trade-off, with several coefficients compared.
Experimental setup
Datasets: SA-1B Multi-Target-SA1B HiMTok training data RefCOCO RefCOCO+ RefCOCOg gRefCOCO
Benchmarks: gRefCOCO RefCOCO RefCOCO+ RefCOCOg RefCOCOm ADE20K (A-150) PASCAL Context59 PASCAL VOC 20 ReasonSeg
Metrics: cIoU gIoU average token length mIoU METEOR CIDEr
Baselines: LISA-7B GSVA-7B HiMTok SAM4MLLM-8B GLaMM PSALM M²SA-7B Text4Seg
Builds on: HiMTok TiTok GRPO InternVL-2.5 UViM
Closest papers by meaning
- SAMTok: Representing Any Mask with Two Words (CVPR 2026 · similarity 0.89)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.88)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.87)
- Adaptive Length Image Tokenization via Recurrent Allocation (ICLR 2025 · similarity 0.87)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.