Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Token Reduction for Large Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MetaCompress, a learned, prompt-agnostic visual token reduction module (a lightweight compression-matrix generator) that outperforms pruning/merging baselines on multi-turn VQA across five LVLMs.

Problem

Visual token reduction for LVLMs is mostly designed for single-turn VQA; in multi-turn VQA future questions are unknown, so prompt-dependent methods overfit to the first prompt and heuristic prompt-agnostic methods (attention-based) perform suboptimally.

Approach

Formulate token reduction as a learnable sparse compression matrix P applied to visual tokens (unifying pruning and merging), and learn a lightweight meta generator that produces P from the image sequence alone via down-sampled query and key inner products, trained with prediction-KL plus entropy and collapse regularizers on a small (~20k) data subset.

Key results

• At 90% reduction, MetaCompress reaches 75.26 average on MT-VQA-v2 for LLaVA-NeXT-13b versus 73.08 for the best baseline (Sample) and 81.66 base.
• On MT-GQA at 90% reduction for LLaVA-1.5-7b, MetaCompress averages 58.43 versus 57.56 for Sample and 45.66 for FastV.
• Efficiency: at 90% reduction on MT-GQA, MetaCompress TTFT and E2E time are comparable to the Sample baseline (e.g., 97.8 ms vs 96.9 ms TTFT on LLaVA-1.5-7b).

Surprising findings

• Learned compression matrices retain tokens that show little correlation with [CLS] or prompt attention, suggesting attention-based heuristics are suboptimal.
• FastV performs worse than random or uniform pruning in the multi-turn setting for most models.

Limitations

• Only the reduction is applied before the LLM decoder; no reduction within LLM layers or the vision tower.
• Training uses a small subset (~20k) and requires per-model training; ConvBench evaluation uses Llama-3.1-8B judge instead of GPT-3.5-turbo.
• Video transfer evaluated only with a single model (XComposer-2.5) on a self-constructed MT-Video-MME with 500 dialogs.

Experimental setup

Datasets: MT-VQA-v2 MT-GQA VQA-v2 GQA ConvBench Video-MME

Benchmarks: MT-VQA-v2 MT-GQA ConvBench MT-Video-MME

Metrics: accuracy per turn average accuracy TTFT E2E generation time GPU memory TFLOPs

Baselines: Random Sample FastV PruMerge Spatial pooling ToMe VisionZip

Builds on: FastV LLaVA-PruMerge FitPrune VisionZip

Closest papers by meaning

• AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition (CVPR 2026 · similarity 0.92)
• Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.92)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.92)
• Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.91)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.91)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.91)