Research map / Vision-language and multimodal / Multimodal LLMs

CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LiteLVLM, a training-free text-guided visual token pruning method for pixel grounding in LVLMs that reverses CLIP visual-text similarity ranking and recovers context tokens via CLS contribution scores.

Problem

Existing visual token pruning methods, built for image understanding, are text-agnostic or use high visual-text similarity, and they fail on pixel grounding (referring expression segmentation and video object segmentation) where referent-relevant tokens matter.

Approach

CLIP's contrastive pretraining causes visual tokens in referent regions to have low similarity to the [EOS] text embedding (a visual-text similarity reversal, aggravated by a text attention sink on [SOS]); LiteLVLM keeps the lowest-similarity tokens and adds back high-[CLS]-contribution context tokens with an adaptive budget split.

Key results

• On RefCOCO/+/g at 192 visual tokens (66.7% pruning), LiteLVLM retains 90.3% of GLaMM's average performance versus 87.4% for VisPruner and 82.9% for FastV.
• On Ref-DAVIS-17 with 196 tokens, LiteLVLM retains 99.5% of VideoGLaMM performance versus 92.5% for VisPruner.
• Efficiency on RefCOCO-val at 192 tokens: CUDA time 265.83 ms vs 340.89 ms for GLaMM (up to 22% faster) and activation memory 0.35 GB vs 0.81 GB.

Surprising findings

• Low visual-text similarity tokens (lowest CLIP similarity to the EOS embedding) concentrate on the referent, the reverse of the usual assumption that high similarity indicates relevance.
• Context-aware-only selection performs worse than random pruning on RefCOCO-val (63.3 vs 65.0 average).

Limitations

• Method relies on CLIP-style [CLS]/[EOS] global tokens; SigLIP-2 requires a heuristic substitute for these tokens.
• Requires fine-tuning to reach the best results; zero-shot retention is lower (e.g., 74.4 vs 77.4 cIoU at 192 tokens).
• Evaluation mainly on GLaMM and VideoGLaMM; generalization to other LVLMs is only tested on UniPixel and X-SAM in appendix.

Experimental setup

Datasets: RefCOCO RefCOCO+ RefCOCOg Ref-DAVIS-17 Refer-YouTube-VOS MeViS

Benchmarks: RefCOCO RefCOCO+ RefCOCOg Ref-DAVIS-17 Refer-YouTube-VOS MeViS VQAv2 GQA TextVQA POPE MMBench MMVet

Metrics: cIoU J&F accuracy

Baselines: TRIM FastV LLaVA-PruMerge VisionZip VisPruner SparseVLM ToMe

Builds on: GLaMM LLaVA-PruMerge TRIM FastV VisPruner

Closest papers by meaning

• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.92)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.92)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.92)
• Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)