Research map / Vision-language and multimodal / Multimodal LLMs
An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune 50% of image tokens after layer 2 by attention rank (FastV) | LLaVA-1.5-13B · Nocaps/Flickr30k/A-OKVQA/MMMU average | average score: 73.6 -> 73.6 | No clear effect |
| increase pruning ratio R from 50% to 90% at K=2 (ablation) | FastV on LLaVA-1.5-7B · Nocaps/Flickr30k/A-OKVQA/MMMU average | average score: 69.7 -> 55.2 | Hurt |
| prune before the LLM (K=0) instead of after layer 2 at 90% ratio (ablation) | FastV on LLaVA-1.5-7B · Nocaps/Flickr30k/A-OKVQA/MMMU average | average score: 55.2 -> 40.4 | Hurt |
| rank image tokens by attention instead of random dropping (ablation) | FastV (K=2, R=50%) · Flickr30k (retrained LLaVA-1.5-7B) | CIDEr: 68.3 -> 70 | Improved |
| prune image tokens with FastV instead of training with 50% fewer image tokens (ablation) | LLaVA-1.5-7B · Nocaps (retrained) | CIDEr: 98.5 -> 100.1 | Improved |
| prune system prompt tokens instead of image tokens (ablation) | FastV · A-OKVQA (retrained LLaVA-1.5-7B) | accuracy: 78.4 -> 69.2 | Hurt |
| prune instruction tokens instead of image tokens (ablation) | FastV · A-OKVQA (retrained LLaVA-1.5-7B) | accuracy: 78.4 -> 56.5 | Hurt |
| apply StreamingLLM sparse attention pattern to LVLM (ablation) | LLaVA-1.5-7B (retrained) · Nocaps | CIDEr: 100.1 -> 13.2 | Hurt |
| apply FastV (K=2, R=50%) to Video-LLaVA | Video-LLaVA · Video QA average | accuracy: 0.48 -> 0.49 | Improved |
| apply FastV (K=2, R=50%) to InstructBLIP-Vicuna-13B | InstructBLIP-Vicuna-13B · MME | total score: 1143.5 -> 1129.8 | Hurt |
| increase filtering layer K from 2 to 5 for InstructBLIP (ablation) | FastV on InstructBLIP-Vicuna-13B · MME | total score: 1129.8 -> 1140.5 | Improved |
| apply FastV (K=0, R=50%) to LLaVA-1.5-13B | LLaVA-1.5-13B · A-OKVQA latency | latency per example: 0.539s -> 0.341s | Improved |
Closest papers by meaning
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.92)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.91)
- Conical Visual Concentration for Efficient Large Vision-Language Models (CVPR 2025 · similarity 0.91)
- A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.91)
- Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.91)
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
