Research map / Vision-language and multimodal / Multimodal LLMs
ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layers
Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select replaced layers by Layer Contribution instead of random choice (ablation) | ShortV (N=19) · MMBench EN | accuracy: 58.4 -> 64.8 | Improved |
| select replaced layers by Layer Contribution instead of cosine similarity (ablation) | ShortV (N=19) · MMBench EN | accuracy: 60.8 -> 64.8 | Improved |
| select replaced layers by Layer Contribution instead of cosine similarity (ablation) | ShortV (N=19) · GQA | accuracy: 59.5 -> 60.9 | Improved |
| freeze visual tokens rather than text tokens (ablation) | ShortV (N=19) · MMBench EN | accuracy: 2.1 -> 64.8 | Improved |
| freeze visual tokens rather than all tokens (ablation) | ShortV (N=19) · GQA | accuracy: 0.0 -> 60.9 | Improved |
| freeze visual tokens rather than the same number of random tokens (ablation) | ShortV (N=19) · SEED-Bench | accuracy: 5.5 -> 66.2 | Improved |
| add FastV on top of ShortV | ShortV on LLaVA-1.5-7B · FLOPs ratio | FLOPs ratio: 55% -> 29% | Improved |
| add FastV on top of ShortV | ShortV on LLaVA-1.5-7B · GQA | accuracy: 60.9 -> 59.3 | Hurt |
| ShortV instead of FastV at similar FLOPs | LLaVA-1.5-7B · Flickr30K | CIDEr: 67.5 -> 71.3 | Improved |
| ShortV instead of VTW at similar FLOPs | LLaVA-1.5-7B · GQA | accuracy: 55.1 -> 60.9 | Improved |
| increase number of ShortV layers from 19 to 24 (ablation) | ShortV on LLaVA-1.5-7B · MME | score: 1503.1 -> 1341.7 | Hurt |
| increase number of ShortV layers from 0 to 24 on LLaVA-1.5-13B (ablation) | ShortV on LLaVA-1.5-13B · 7-benchmark retention | relative performance (Per.): 100.0 -> 99.2 | No clear effect |
Closest papers by meaning
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.92)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.91)
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.91)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.91)
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
