Research map / Vision-language and multimodal / Multimodal LLMs

Grounding Everything in Tokens for Multimodal Large Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GETok, which adds learnable 2D grid tokens and offset tokens (with a DELETE token) to an MLLM's vocabulary for grounding points, boxes, polylines and masks, trained with SFT and a two-stage GRPO scheme on Qwen2.5-VL-7B.

Problem

MLLMs struggle to ground objects precisely in 2D image space because text coordinates, patch tokens, and 1D bin tokens do not preserve 2D spatial topology, and initial localization errors are rarely corrected.

Approach

Discretize the image into an n×n grid where each cell is a learnable vocabulary token (spatial anchor), plus a small set of offset tokens (displacements in {-1,0,1}² and DELETE) for coarse-to-fine iterative refinement; masks are converted to sets of grid tokens by a training-free greedy IoU algorithm over SAM masks; RL uses geometry-aware rewards in a two-stage GRPO.

Key results

• GETok-R1 reaches 72.7 average on RES (ReasonSeg, RefCOCO+/g) vs 70.7 for VisionReasoner, and 90.3 Acc@0.5 average on REC vs 88.7 for VisionReasoner.
• GETok-SFT achieves 88.2 average Acc@0.5 on REC vs 86.6 for Qwen2.5-VL-7B, and 72.8 vs 70.9 at Acc@0.8.
• On the driving dataset GETok improves traffic sign color recognition from 71.43 to 83.67 over visual prompts; lane detection F1 0.58 vs 0.48 for coordinate-based.

Surprising findings

• Offset tokens add only 10 tokens yet a 32x32 grid with offsets (72.6 REC Acc@0.8) outperforms a 64x64 grid (71.2) which has far more tokens.
• Reward designs without a proper signal led the model to predict no offsets at all, collapsing refinement.

Limitations

• Real-world driving dataset is proprietary and not released, limiting reproducibility of the case study.
• Gains on RefCOCO are modest because expressions are simple, leaving less room for RL refinement.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg ReasonSeg LISA++ gRefCOCO COCO-Stuff ADE20K Visual Genome PACO-LVIS PASCAL-Part Pixmo-point

Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReasonSeg gRefCOCO Visual Genome HumanRef Pixmo-point

Metrics: Acc@0.5 Acc@0.8 gIoU cIoU METEOR CIDEr precision recall F1

Baselines: LISA PixelLM LAVT ReLA CRIS Qwen2.5-VL-7B Seg-Zero SAM-R1 VisionReasoner UNINEXT-L Shikra Ferret

Builds on: Qwen2.5-VL LISA GRPO VisionReasoner Seg-Zero SAM

Closest papers by meaning

• See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Model (NeurIPS 2025 · similarity 0.91)
• GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.90)
• Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.90)
• Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.90)
• Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models (NeurIPS 2025 · similarity 0.89)
• Grounding Multimodal Large Language Models in Actions (NeurIPS 2024 · similarity 0.89)