Research map / Vision-language and multimodal / Multimodal LLMs

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

AAAI 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 23 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FitPrune, a training-free visual token pruning method that uses binary search over attention statistics from a small batch of data to find a layer-wise pruning recipe meeting a FLOPs budget.

Problem

Visual tokens in MLLMs are redundant and expensive, but deciding when and how many to prune per layer for a given computation budget normally requires costly manual trials.

Approach

Treat token pruning as distribution fitting: choose per-layer token removal counts so that the self-attention and cross-attention distributions over visual tokens before and after pruning diverge minimally, found via binary search on a divergence threshold using attention statistics from ~655 samples.

Key results

• On LLaVA-NEXT 7B, FitPrune at a 60% visual-FLOPs target reduces total TFLOPs by about 54-55% with average accuracy 66.8 vs 67.1 on five benchmarks.
• On LLaVA-1.5 7B at 60% pruning, FitPrune averages 63.2 accuracy vs 63.3 baseline with about 48-50% TFLOPs reduction.
• Pruning recipe for LLaVA-1.5 is obtained in about 5 minutes using 655 samples; GQA accuracy stays ~61.6 even with only 65 samples.

Measured effects reported in the paper

Surprising findings

• Using only 65 samples to build the pruning recipe gives nearly the same GQA accuracy as using 6.6K samples, suggesting the attention pattern depends mainly on the model itself.
• Self-attention-only or cross-attention-only fitting degrades substantially (e.g., self-only GQA 55.4 vs 62.0), while combining both recovers baseline accuracy.

Limitations

• Search is tied to a FLOPs budget and attention statistics from a sample set; results depend on the calibration data.
• Reported results on a limited set of benchmarks, and some numbers are inconsistent across text and tables (e.g., 54.9% vs 54.1% TFLOPs reduction, FLOPs drop claims).

Experimental setup

Datasets: LLaVA-655k

Benchmarks: ScienceQA-IMG GQA TextVQA VizWiz VQAv2 POPE MM-Vet MMBench MMBench-CN MME

Metrics: accuracy TFLOPs throughput

Baselines: FastV ToMe MADTP Zero-TPrune ATS DiffRate

Builds on: FastV ToMe LLaVA-1.5

Closest papers by meaning

• What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.94)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.94)
• Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.94)
• ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.93)
• Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.93)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)