Research map / Vision-language and multimodal / Multimodal LLMs

IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IF-Prune, which fine-tunes a small InternVL2.5-1B with a token-wise variational information bottleneck and uses per-token KL-to-prior as an importance score to prune visual tokens for a larger VLM in a single forward pass.

Problem

Visual token pruning for dynamic-resolution VLMs, where existing small-VLM guidance (SGP) relies on answer-driven attention and is unreliable at high pruning ratios.

Approach

Instead of aggregating attention from generated answers, a small VLM is fine-tuned with a token-level variational information bottleneck so that each visual token's KL divergence from a learnable prior, conditioned on the query, serves as its importance score; this guides hard pruning of a larger VLM with the same architecture.

Key results

• With 5% of visual tokens retained on InternVL2-26B, IF-Prune keeps 95.41% of the original score versus 88.50% for SGP and 75.05% for FastV.
• With 20% retention, IF-Prune reaches 99.55% score ratio on InternVL2-26B, versus 99.15% for SGP.
• Guide model InternVL2.5-1B transfers to InternVL2-8B: at 5% retention IF-Prune gets 94.03% vs 90.34% for SGP; total FLOPs reduced to 62.9% of baseline at 5% retention.

Surprising findings

• Prefill latency with IF-Prune (238.5 ms) is close to the unpruned baseline (229.1 ms), while SGP's is 524.5 ms.

Limitations

• Small VLM must share architecture with the large VLM for consistent visual encoding.
• Large VLM FLOPs increase slightly due to longer generated responses in some settings.
• Evaluated only on InternVL family and single-image tasks.

Experimental setup

Datasets: ShareGPT-4V LLaVA instruction data DVQA

Benchmarks: TextVQA ChartQA GQA MMStar MMBench MM-Vet MME RealWorldQA

Metrics: accuracy score ratio FLOPs prefill latency decode latency throughput

Baselines: FastV SGP ToMe

Builds on: variational information bottleneck SGP FastV InternVL

Closest papers by meaning

• QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.95)
• LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.93)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass (NeurIPS 2026 · similarity 0.93)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity (ICLR 2026 · similarity 0.92)