Research map / Vision-language and multimodal / Multimodal LLMs
Efficient Segmentation with Multimodal Large Language Model via Token Routing
Changsong Wen, Zelin Peng, Yu Huang, Wei Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an efficient MLLM-based segmentation method that routes tokens to reduce LLM computation, rather than pruning visual tokens.
Problem
The LLM component of MLLMs is computationally expensive for open-world segmentation, where efficiency is a central concern; existing efficient MLLM methods mainly prune visual tokens.
Approach
Uses token routing to reduce LLM computation for segmentation instead of pruning visual tokens, as an alternative to existing visual token pruning approaches.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| route mask tokens only from layer 13 of the LLM (skip early layers) (ablation) | PSALM (Phi-1.5) · RefCOCO | cIoU: 81.56 -> 81.65 | No clear effect |
| delay mask token insertion further to layer 18 (ablation) | Token Routing · RefCOCO | cIoU: 81.65 -> 80.48 | Hurt |
| skip the LLM entirely for mask tokens (ablation) | Token Routing · RefCOCO | cIoU: 81.65 -> 79.93 | Hurt |
| skip FFN for all image tokens (ablation) | PSALM (Phi-1.5) · RefCOCO | cIoU: 81.71 -> 80.40 | Hurt |
| add learnable gate to proxy-guided update (ablation) | Proxy-guided image routing · RefCOCO | cIoU: 81.02 -> 81.71 | Improved |
| increase proxy tokens from 8 to 32 (ablation) | Proxy-guided image routing · RefCOCO | cIoU: 81.71 -> 81.65 | No clear effect |
| prune image tokens (FastV, 50%) instead of routing | PSALM (Phi-1.5) · RefCOCO series average | cIoU: 69.2 -> 66.9 | Hurt |
| apply late-stage mask token routing alone (ablation) | PSALM (Phi-1.5) · RefCOCO series average | FLOPs (G): 1126.98 -> 998.32 | Improved |
| apply proxy-guided image token routing alone (ablation) | PSALM (Phi-1.5) · RefCOCO series average | FLOPs (G): 1126.98 -> 758.75 | Improved |
| combine both routing strategies | PSALM (Phi-1.5) · RefCOCO series average | latency (ms): 123 -> 83 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- LLaVA-Mini — LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025) · better on 1 of 1 shared result
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.91)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.90)
- SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs (NeurIPS 2025 · similarity 0.90)
- What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
