Research map / Vision-language and multimodal / Multimodal LLMs
ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ATP-LLaVA, which learns instance- and layer-specific visual token pruning thresholds inside LLaVA-1.5's LLM decoder, combining redundancy and spatial pruning scores with a differentiable budget-constrained loss.
Problem
Fixed or pre-defined visual token pruning ratios in large vision language models are suboptimal because the effect of pruning varies across LLM layers and input instances, while visual token processing is computationally expensive.
Approach
An Adaptive Token Pruning module inserted between any two LLM layers predicts instance-specific redundancy and spatial pruning thresholds from self- and cross-modal attention scores; soft sigmoid masks make the thresholds trainable, and a Spatial Augmented Pruning strategy keeps spatial structure via uniform sampling and original positional embeddings.
Key results
- With an average of 144 tokens (from 576), ATP-LLaVA retains 98.1% of LLaVA-1.5 performance across seven benchmarks, with 75% token reduction.
- At an average of 88 tokens, ATP-LLaVA retains 94.6% performance; it reduces FLOPs by 78.1% and CUDA time by 38.4% at 144 tokens.
- Under the same average token budget, ATP-LLaVA outperforms fixed-ratio pruning by 2.8 (MMB), 1.6 (VQAv2) and 1.6 (SEED) points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace fixed pruning ratios with instance- and layer-adaptive pruning at the same average token budget | ATP-LLaVA · MMBench | score: 62.7 -> 66.0; 2.8 | Helped |
| Replace fixed pruning ratios with instance- and layer-adaptive pruning at the same average token budget | ATP-LLaVA · VQA v2 | score: 74.6 -> 76.4; 1.6 | Helped |
| Replace fixed pruning ratios with instance- and layer-adaptive pruning at the same average token budget | ATP-LLaVA · SEED-Bench | score: 55.7 -> 57.3; 1.6 | Helped |
| Fine-tune the language model rather than freeze it while training ATP (ablation) | ATP-LLaVA · MMBench | score: 63.1 -> 66.0 | Helped |
| Add redundant pruning (ablation) | ATP-LLaVA · MMBench | score: 65.3 -> 66.0 | Helped |
| Add spatial pruning (ablation) | ATP-LLaVA · GQA | score: 57.1 -> 59.5 | Helped |
| Add pruning positional embedding to spatial pruning (ablation) | ATP-LLaVA · MMBench | score: 64.3 -> 65.2 | Helped |
| Combine self-modality and cross-modality importance scores instead of using self-score only (ablation) | ATP-LLaVA redundant pruning · GQA | score: 58.7 -> 59.5 | Helped |
| Combine self-modality and cross-modality importance scores instead of using cross-score only (ablation) | ATP-LLaVA redundant pruning · SEED-Bench | score: 56.3 -> 57.3 | Helped |
| Prune to an average of 144 tokens | ATP-LLaVA · seven vision-understanding benchmarks | performance retention: 98.1{ \%} | Hurt |
| Prune to an average of 88 tokens | ATP-LLaVA · seven vision-understanding benchmarks | performance retention: 94.6{ \%} | Hurt |
Surprising findings
- ATP-LLaVA slightly exceeds the LLaVA-1.5 upper bound on MMBench (66.0 vs 64.3) at 144 average tokens.
- Freezing the LLM and training only the ATP module yields lower performance (e.g., GQA 57.9 vs 59.5), indicating LM fine-tuning matters for adapting to pruned tokens.
Limitations
- Relies on a token-count penalty weight and target-budget hyperparameters that must be tuned per budget.
- Training uses soft masks during training and hard discarding at inference, so train and inference behaviors differ.
- Evaluated only on LLaVA-1.5 with Vicuna-7B and image benchmarks.
Experimental setup
Datasets: LLaVA-1.5 665k visual instruction data CC3M (filtered, for pretrained projector)
Benchmarks: GQA MMBench MME POPE SEED-Bench ScienceQA-IMG VQAv2
Metrics: accuracy retention rate relative to upper bound average token count FLOPs CUDA time KV cache storage memory
Baselines: LLaVA-1.5 LLaVA-PruMerge+ FastV SparseVLM
Builds on: LLaVA-1.5 FastV LLaVA-PruMerge SparseVLM RoPE
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.95)
- LearnPruner: Rethinking Attention-based Token Pruning in Vision Language Models (ICLR 2026 · similarity 0.94)
- An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.93)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.93)
- QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models (CVPR 2026 · similarity 0.93)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.93)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.