Research map / Vision-language and multimodal / Multimodal LLMs

ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ClawMachine, an MLLM that references and grounds objects by emitting groups of quantized visual tokens ("token collectives") in a joint vision-language vocabulary, with hybrid continuous/discrete perception and no extra location syntax.

Problem

Fine-grained referring and grounding in MLLMs currently needs extra syntax (proxy tokens, text coordinates, or location bins) or extra vision modules, which complicates vision-language alignment, training, and inference efficiency.

Approach

Represent a referred region as the set of quantized visual tokens (from LaVIT's VQ codebook) whose patch centers fall inside its box, inserted directly into the text sequence, so referring and grounding become next-token prediction in a joint vocabulary; combine continuous CLIP features (via MLP) with discrete tokens (via LLM embedding) as hybrid perception.

Key results

• ClawMachineX-7B reaches 118.4 CIDEr on RefCOCOg referring and 169.7 CIDEr on Visual Genome, above Groma-7B (107.3 and 158.4).
• ClawMachineX-7B achieves 89.7 / 92.5 / 86.9 on RefCOCO val/test-A/test-B grounding accuracy, the best among listed MLLMs, with Groma-7B (89.5 / 92.1 / 86.2) the closest.
• On Ref-Hal, ClawMachine-7B reaches 75.8 accuracy versus 61.5 for the next-best baseline (Qwen-VL-7B).

Measured effects reported in the paper

Surprising findings

• Hybrid perception (continuous plus discrete tokens) gives 85.7 REC versus 73.1 with continuous features only in the ablation table.
• Pre-training on GRIT-20M with 15M pairs yields a better grounding score than the 700K instruction-tuning data alone, indicating scaled pre-training matters more than heavy instruction tuning.

Limitations

• Quantizing visual features loses fine-grained detail.
• The LLM's probabilistic output can yield unstable or incomplete token collectives that miss parts of the target.
• Grounding boxes depend on a separate region sampler with an external detection proposer at inference.

Experimental setup

Datasets: LLaVA-Pretrain ShareGPT-4V RefCOCO/+/g Visual Genome GRIT-20M LLaVA-mix665k Osprey AS-V2 Chatterbox GQA LVIS

Benchmarks: RefCOCOg Visual Genome RefCOCO/+/g LVIS-Ground POPE Ref-Hal LLaVA-Bench (COCO) VQAv2

Metrics: METEOR CIDEr accuracy@IoU0.5 AR accuracy F1 precision recall

Baselines: Shikra-7B Ferret-7B GLaMM-7B Groma-7B Qwen-VL-7B MiniGPT-v2 Florence-2 Osprey-7B GPT4RoI Kosmos-2 Groundhog-7B LLaVA-7B

Builds on: LaVIT LLaVA EVA-CLIP Groma Shikra

Closest papers by meaning

• Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs (CVPR 2026 · similarity 0.91)
• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
• UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.90)
• Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
• Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.90)
• Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.90)