Research map / Vision-language and multimodal / Multimodal LLMs

Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that visual token compression collapses on MLLM-based referring expression segmentation because position embeddings matter, and proposes PAYN, a training-free, position-only token selection (checkerboard or farthest point sampling) that keeps original position indices.

Problem

Token compression methods that work on vision-language tasks degrade severely (retaining only ~20% of performance) when applied to MLLM-based referring expression segmentation, which is rarely studied.

Approach

Dense RES is highly sensitive to local spatial structure and original position indices, so tokens can be selected purely by spatial dispersion (checkerboard sampling or farthest point sampling in 2D coordinates) while keeping their original positional IDs, requiring no features or attention.

Key results

• On Text4Seg (LLaVA-1.5-7B) with 192 of 576 tokens, PAYN reaches 66.4 average cIoU (88.6% of vanilla 74.9), versus 63.6 for DivPrune and 12.9 for ToMe.
• On InstructSeg with 72 of 729 tokens (90% reduction), PAYN averages 71.7 cIoU (87.6% retained) vs 70.2 for EVTP-IVS and 69.9 for DivPrune.
• Removing preserved position IDs drops PAYN from 66.4 to 10.2 average cIoU on Text4Seg with 192 tokens.

Surprising findings

• Token pruning with contiguous re-indexed positions collapses to about 10% cIoU on RES, while the same selection with original position IDs recovers over 50 points.
• Pre-encoder patch dropping causes complete collapse (avg 9.9 cIoU), indicating discarded patches cannot be recovered after the vision encoder.

Limitations

• Can fail for extremely small objects when very few tokens are retained, since the selection ignores content.
• Background with little informative content and extremely low token budgets may lead to suboptimal local structure preservation.
• Compression ratios are tuned per baseline and results are reported mainly on RES benchmarks with two baselines.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg MME ScienceQA TextVQA

Benchmarks: RefCOCO RefCOCO+ RefCOCOg

Metrics: cIoU relative performance retained

Baselines: ToMe VisionZip Dart VisPruner DivPrune EVTP-IVS PruMerge

Builds on: Text4Seg InstructSeg DivPrune VisionZip

Closest papers by meaning

• Unveiling Parts Beyond Objects: Towards Finer-Granularity Referring Expression Segmentation (CVPR 2024 · similarity 0.88)
• PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
• 3D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentation (AAAI 2024 · similarity 0.87)
• Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.87)
• Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.86)
• CORE: Compact Object-centric REpresentations as a New Paradigm for Token Merging in LVLMs (CVPR 2026 · similarity 0.86)