Research map / Vision-language and multimodal / Multimodal LLMs

Reasoning to Attend: Try to Understand How <SEG> Token Works

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 10 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how the <SEG> token in LISA-style LMMs relates to image tokens via similarity maps, then proposes READ, which turns highly activated similarity points into differentiable SAM prompts (Similarity as Points) to improve reasoning segmentation and referring segmentation.

Problem

Segmentation LMMs such as LISA rely on a <SEG> token to pass information from an LLaVA-based VLM to a SAM mask decoder, but little is known about what this token computes. The paper asks how it relates semantically to image tokens and whether that can guide reasoning segmentation.

Approach

Compute a parameter-free dot-product similarity between the <SEG> hidden state and image token hidden states, select highly activated positive/negative points, and convert them into continuous differentiable coordinates via Gaussian-like weighted interpolation (Discrete-to-Continuous) so they can be fed as extra prompts to SAM and gradients flow back to the LMM.

Key results

• READ-7B-LLaVA1.5 (ft) reaches 58.5 gIoU / 58.6 cIoU overall on ReasonSeg test, versus 55.6/56.9 for LISA-7B-LLaVA1.5 (ft) reported in the paper's table.
• READ-13B reaches 62.2 gIoU / 62.8 cIoU overall on ReasonSeg test, versus 61.3/62.2 for LISA-13B-LLaVA1.5 (ft).
• On the ablation, adding similarity points (P_prompt) to the <SEG> prompt raises cIoU from 57.6 to 64.6 on ReasonSeg val, and DtoC further raises it to 67.6.

Measured effects reported in the paper

Surprising findings

• Points selected purely from the similarity map, without the <SEG> token, can yield a SAM mask with cIoU of 27.0% on ReasonSeg test, close to the 30.4% obtained by the <SEG> prompt (SESAME).
• The LLaVA encoder's textual reasoning can remain accurate even when the SAM decoder fails to segment the target object.

Limitations

• The similarity computation is a simple parameter-free dot product; the authors acknowledge more effective learnable similarity (e.g., cross attention) is left for future work.
• Analysis is largely qualitative and based on a small set of examples and a single grid-searched threshold setup.
• READ-7B is initialized from the released SESAME checkpoint, and comparisons depend on reproduced baselines and different training data ratios.

Experimental setup

Datasets: ReasonSeg ADE20K COCO-Stuff PACO-LVIS PASCAL-Part refCLEF refCOCO refCOCO+ refCOCOg LLaVA-Instruct-150k FP-RefCOCO(+/g) R-RefCOCO

Benchmarks: ReasonSeg RefCOCO RefCOCO+ RefCOCOg FP-RefCOCO FP-RefCOCO+ FP-RefCOCOg

Metrics: gIoU cIoU see accuracy

Baselines: LISA-7B SESAME LISA-7B-LLaVA1.5 (ft) GSVA GLaMM Grounded-SAM SEEM OVSeg X-Decoder GRES CRIS LAVT

Builds on: LISA SESAME LLaVA SAM

Closest papers by meaning

• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.87)
• Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation (AAAI 2025 · similarity 0.87)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.87)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.87)
• ViCToR: Improving Visual Comprehension via Token Reconstruction for Pretraining LMMs (AAAI 2026 · similarity 0.87)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.87)