Research map / Vision-language and multimodal / Multimodal LLMs
SAMTok: Representing Any Mask with Two Words
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAMTok, a SAM-based residual-quantized mask tokenizer that maps any region mask to two discrete tokens, letting QwenVL models learn pixel-wise understanding and generation via next-token SFT and a text-only GRPO reward.
Problem
Pixel-wise MLLMs are hard to scale because they need region-level encoders, specialized segmentation decoders, and task-specific losses, while text- or polygon-based mask representations produce too many tokens.
Approach
Treat masks as a new language: a SAM-initialized encoder plus two-step residual quantizer compresses each mask into two special tokens that a SAM decoder reconstructs; MLLMs then learn masks via plain next-token prediction and are RL-trained with a string-matching reward on mask tokens.
Key results
- On GRES, Qwen2.5-VL-SAMTok after GRPO gains +8.9% gIoU and +21.0% N-acc on the validation set over the SFT model.
- Qwen2.5VL-SAMTok (3B, rl) reaches 73.2 mIoU and 53.5 Recall on GCG test, and the 7B model with SFT is best on several GCG metrics.
- On DLC-Bench, Qwen3VL-SAMTok (4B) achieves 65.6 average vs. 67.3 for the expert DAM (3B), while beating it on negative-prompt accuracy (85.2 vs 82.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| GRPO reinforcement learning with textual answer-matching reward on mask tokens | Qwen2.5-VL-3B SAMTok · GCG val | AP50 / Recall: +4.7% AP50 and +6.6% Recall average gain (paper text) | Helped |
| GRPO reinforcement learning with textual answer-matching reward | Qwen2.5-VL-3B SAMTok · GRES val | gIoU / N-acc: +8.9% gIoU and +21.0% N-acc (paper text) | Helped |
| Adding unified mask-token interface instead of text-box/point grounding | Qwen2.5-VL 3B and 7B on RefCOCO grounding · RefCOCO/+/g REC | accuracy: +3.6 (RefCOCO val, 3B) and +3.0 (7B) | Helped |
| Residual quantization (RQ, 1024x2) instead of standard VQ at 65536 codebook (ablation) | SAMTok quantizer · EntitySeg reconstruction / RefCOCO generation | r-Acc and g-Acc: r-Acc 0.70 vs 0.66; g-Acc 78.3 vs 77.8 | Helped |
| Increasing residual quantization steps from 2 to 4 (1024x4 vs 1024x2) (ablation) | SAMTok quantizer · EntitySeg reconstruction / RefCOCO generation | r-Acc and g-Acc: r-Acc 0.75 vs 0.70 (better); g-Acc 77.3 vs 78.3 (worse) | Mixed |
| Increasing codebook size under 2 residual steps (256x2 to 512x2) (ablation) | SAMTok quantizer · EntitySeg reconstruction / RefCOCO generation | r-Acc and g-Acc: r-Acc 0.70 to 0.71; g-Acc 77.6 to 77.8 | No clear effect |
| Training SAMTok with the MLLM (joint training) vs decoupled tokenizer | Mask tokenizer reconstruction · Qualitative mask reconstruction | reconstruction quality | Hurt |
| Adding mask tokens via SAMTok for region captioning vs general Qwen2.5-VL | Qwen2.5-VL 7B on DLC-Bench · DLC-Bench | Avg.: 41.2 (Qwen2.5VL) vs 65.6 (Qwen3VL-SAMTok 4B) | Helped |
| Mask-token interface for PSG generation with Qwen2.5-VL 7B | Qwen2.5-VL 7B on PSG · PSG R@20 | R@20 / mR@20: R@20 19.8 vs VCTree 20.6; mR@20 15.4 vs 14.8 (PSGFormer) | No clear effect |
| SFT with SAMTok mask tokens across MLLM backbones (Qwen2.5-VL 3B, Qwen3-VL 4B, PLM 1B) (ablation) | GCG and GRES generation · GCG / GRES averaged | AP50 / gIoU: PLM-SAMTok AP50 41.9 and gIoU 74.7 vs Qwen25VL-SAMTok 36.4 and 70.1 | Helped |
Surprising findings
- Joint-training mask tokenizers collapse to elliptical masks, while the decoupled SAMTok keeps reconstruction quality independent of downstream MLLM training.
- The compressed two-token mask representation outperforms text box grounding on RefCOCO-family benchmarks, with gains of up to +10.3 points over Qwen2.5-VL.
Limitations
- Only handles 2D image region masks and cannot reconstruct masks in video.
- Only covers mask representation; points, lines, and boxes are not included in the interface.
- Longer mask words (more residual steps) increase generation difficulty and computational cost for dense masks.
Experimental setup
Datasets: EntitySeg GRES GCG RefCOCO/RefCOCO+/RefCOCOg MR-RefCOCO MR-PACO PSG DLC-Bench MDVP-Bench VideoRefer-D GroundingSuite
Benchmarks: GCG GRES RefCOCO/RefCOCO+/RefCOCOg GroundingSuite MR-RefCOCO/+/g MR-PACO DLC-Bench MDVP-Bench VideoRefer-D PSG EntitySeg
Metrics: METEOR CIDEr AP50 mIoU Recall gIoU cIoU N-acc cIoU per round R@20
Baselines: LISA GLaMM OMG-LLaVA Sa2VA SegLLM SAM4MLLM MLLMSeg HiMTok ARGenSeg DAM PAM Osprey
Builds on: SAM 2 GRPO Qwen2.5-VL residual vector quantization LISA
Closest papers by meaning
- ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation (NeurIPS 2025 · similarity 0.89)
- Hybrid Token Compression for Vision-Language Models (CVPR 2026 · similarity 0.87)
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.86)
- PROXSPARSE: REGULARIZED LEARNING OF SEMI-STRUCTURED SPARSITY MASKS FOR PRETRAINED LLMS (ICML 2025 · similarity 0.86)
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.86)
- Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.