Research map / Vision-language and multimodal / Multimodal LLMs
Attention-based visual token pruning
Drops redundant visual tokens inside VLMs using attention or text-guided importance scores to speed up inference.
117 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2024–2026) · an approach in Multimodal LLMs, keyword cluster “tokens · pruning · visual token”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 2 tracked benchmarks.
4 papers in 2023–24 and 113 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 1.9% to 5.2%.
Open questions the data points to
- MMBench (score): best went from 64.3 (up to 2024) to 75.7 in 2025 — HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models (CVPR 2025 · 5 citations)
- GQA (accuracy) is new: first reported in 2025–26 by 4 papers.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2024 · Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · 10 citations)
- 2025 · VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025 · 46 citations)
- 2026 · Remember Me: Bridging the Long-Range Gap in LVLMs with Three-Step Inference-Only Decay Resilience Strategies (AAAI 2026 · 3 citations)
Earliest papers here
- Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · 10 citations)
- Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images (ICLR 2024 · 4 citations)
- Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · 4 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
MMBench · score (higher is better)
5 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 64.3 | Matryoshka Query Transformer for Large Vision-Language Models (NeurIPS 2024 · 4 citations) |
| 2025 | 75.7 | HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models (CVPR 2025 · 5 citations) |
GQA · accuracy (higher is better)
4 papers, 2025–2026 · first reported in 2025–26
| Year | Best so far | Set by |
|---|---|---|
| 2025 | 57 | VoCo-LLaMA: Towards Vision Compression with Large Language Models (CVPR 2025 · 13 citations) |
| 2026 | 58.3 | Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models (ICML 2026 · 0 citations) |
What has been tried
35 papers here report 409 measured changes (a component added, removed or swapped, compared with a baseline): helped 67%, no clear effect 9%, hurt 22%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add efficient projector fine-tuning with 1/10 of the LLaVA-1.5 dataset | LLaVA-1.5 image understanding benchmarks · average relative accuracy | 94.0% -> 95.2% | Helped | VisionZip: Longer is Better but Not Necessary in Vision Language Models (CVPR 2025 · 46 citations) |
| scale spatial resolution with dynamic tiling | DocVQA · score | 40.6 -> 91.1 | Helped | NVILA: Efficient Frontier Visual Language Models (CVPR 2025 · 37 citations) |
| Add multi-scale features with AvgPool | Avg-5 · score | 62.6 -> 62.7 | Helped | FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · 27 citations) |
| Fine-tune the language model rather than freeze it while training ATP | MMBench · score | 63.1 -> 66.0 | Helped | ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · 18 citations) |
| CLS-attention-based budget allocation (layer 0) vs even budget split | TextVQA / DocVQA · accuracy | TextVQA 52.2→54.8; DocVQA 48.8→59.2 | Helped | HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models (AAAI 2025 · 15 citations) |
| Increase number of VoCo tokens from 1 to 128 | 7 image benchmarks · compression retention rate | 83.8% at 1 token to 97.7% at 128 tokens | Helped | VoCo-LLaMA: Towards Vision Compression with Large Language Models (CVPR 2025 · 13 citations) |
| add uniform token recovery after pruning | OCRBench · score | 29.7 -> 30.5 | Helped | TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · 11 citations) |
| add cross-layer dependency mining | ScienceQA, W4A4 · accuracy | 77.53 -> 78.66 | Helped | Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · 10 citations) |
| Aggregate attention from all layers of small 2B model rather than a single layer | TextVQA, 9% token retention · score | 78.98 vs 43.84 (one layer of 26B) | Helped | A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · 7 citations) |
| Replace raw OCR words with DocVLM OCR encoding | DocVQA test · ANLS | 85.8 -> 89.4; 84.4 -> 89.2; 89.1 -> 91.9 | Helped | DocVLM: Make Your VLM an Efficient Reader (CVPR 2025 · 5 citations) |
| add search space evolution | ScienceQA · accuracy | 81.07 -> 83.05 | Helped | EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · 5 citations) |
| Add SRA global and local fusion to SMS | VQA-Text · score | 60.0 -> 61.5 | Helped | HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models (CVPR 2025 · 5 citations) |
What reviewers will expect
Datasets and baselines named in the 113 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- TextVQA · 33 of 113 papers
- MME · 29 of 113 papers
- POPE · 29 of 113 papers
- GQA · 28 of 113 papers
- ScienceQA · 25 of 113 papers
- MMBench · 25 of 113 papers
- VQAv2 · 22 of 113 papers
- VizWiz · 14 of 113 papers
Baselines
- FastV · 58 of 113 papers
- SparseVLM · 37 of 113 papers
- VisionZip · 34 of 113 papers
- ToMe · 22 of 113 papers
- DivPrune · 19 of 113 papers
- PDrop · 18 of 113 papers
- DART · 14 of 113 papers
- PyramidDrop · 13 of 113 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 117 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 9 papers
- Tencent · 4 papers
- Amazon · 4 papers
- Shanghai Jiao Tong University · 4 papers
- University of Science and Technology of China · 3 papers
- The Chinese University of Hong Kong · 3 papers
- Hong Kong University of Science and Technology · 3 papers
- Huawei · 3 papers
- Huazhong University of Science and Technology · 3 papers
- National University of Singapore · 3 papers
Use it
Download BibTeX of 20 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Training-free token reduction for MLLMs (90 papers)
Elsewhere in the topic
- Chain-of-thought multimodal reasoning (332 papers)
- VLM capability analysis (264 papers)
- Multimodal LLM benchmarks (252 papers)
- Spatial reasoning in VLMs (207 papers)
- Instruction-tuned multimodal assistants (199 papers)
- Benchmarks and analysis of MLLMs (181 papers)
- Reinforcement learning for visual reasoning (163 papers)
- Interleaved image-text generation with MLLMs (153 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).