Research map / Vision-language and multimodal / Multimodal LLMs

Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RESTORE, a visual token reduction framework that retains original RoPE position indices with a distance-aware attention calibration term and selects distinctive anchors for token merging, improving text-agnostic VTR baselines on LLaVA and Qwen2.5-VL.

Problem

Visual token reduction in MLLMs distorts the LLM's attention: reduced visual tokens lose total attention mass due to softmax normalization and RoPE long-distance decay, and merged anchor tokens often poorly represent their groups, causing information loss.

Approach

Keep original position indices of retained tokens and multiply attention by a distance-aware calibration term (log token size times (c minus RoPE decay)) to counteract RoPE positional bias; plus select merging anchors by weighting representativeness with discriminativeness computed from the existing self-correlation matrix.

Key results

• With HoloV at 64 retained tokens on LLaVA-1.5-7B, RESTORE reaches 96.5% of full-token average, versus 92.2% for HoloV alone (+4.3%).
• At 192 retained tokens, RESTORE with VisPruner and HoloV retains over 98% of the original average performance.
• On Qwen2.5-VL-7B at 11.1% retention, HoloV improves from 83.5% to 88.4% with RESTORE.

Surprising findings

• Applying attention calibration alone (without retaining original positions) or with reindexed positions degrades accuracy (e.g., 92.2% to 91.0%), and calibration on reindexed positions over-amplifies visual attention.
• Merging regresses TextVQA performance for some baselines at 192 tokens, attributed to feature averaging dilution of localized text details.

Limitations

• Merging can hurt fine-grained tasks such as TextVQA, where localized details are diluted by feature averaging.
• Hybrid pruning-merging gains depend on pruning ratio; multi-stage methods like FlowCut may accumulate distortion from repeated merging.

Experimental setup

Datasets: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA SEED-Bench OCRBench

Benchmarks: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA SEED-Bench OCRBench

Metrics: accuracy average relative performance vs. full tokens latency FLOPs weighted Hausdorff distance

Baselines: FastV PDrop SparseVLM ToMe VisionZip DivPrune VisPruner HoloV FlowCut

Builds on: VisionZip ToMe HoloV FastV density peak clustering

Closest papers by meaning

• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.93)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.92)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.91)