AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 362 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
prune 50% of image tokens after layer 2 by attention rank (FastV)LLaVA-1.5-13B · Nocaps/Flickr30k/A-OKVQA/MMMU averageaverage score: 73.6 -> 73.6No clear effect
increase pruning ratio R from 50% to 90% at K=2 (ablation)FastV on LLaVA-1.5-7B · Nocaps/Flickr30k/A-OKVQA/MMMU averageaverage score: 69.7 -> 55.2Hurt
prune before the LLM (K=0) instead of after layer 2 at 90% ratio (ablation)FastV on LLaVA-1.5-7B · Nocaps/Flickr30k/A-OKVQA/MMMU averageaverage score: 55.2 -> 40.4Hurt
rank image tokens by attention instead of random dropping (ablation)FastV (K=2, R=50%) · Flickr30k (retrained LLaVA-1.5-7B)CIDEr: 68.3 -> 70Improved
prune image tokens with FastV instead of training with 50% fewer image tokens (ablation)LLaVA-1.5-7B · Nocaps (retrained)CIDEr: 98.5 -> 100.1Improved
prune system prompt tokens instead of image tokens (ablation)FastV · A-OKVQA (retrained LLaVA-1.5-7B)accuracy: 78.4 -> 69.2Hurt
prune instruction tokens instead of image tokens (ablation)FastV · A-OKVQA (retrained LLaVA-1.5-7B)accuracy: 78.4 -> 56.5Hurt
apply StreamingLLM sparse attention pattern to LVLM (ablation)LLaVA-1.5-7B (retrained) · NocapsCIDEr: 100.1 -> 13.2Hurt
apply FastV (K=2, R=50%) to Video-LLaVAVideo-LLaVA · Video QA averageaccuracy: 0.48 -> 0.49Improved
apply FastV (K=2, R=50%) to InstructBLIP-Vicuna-13BInstructBLIP-Vicuna-13B · MMEtotal score: 1143.5 -> 1129.8Hurt
increase filtering layer K from 2 to 5 for InstructBLIP (ablation)FastV on InstructBLIP-Vicuna-13B · MMEtotal score: 1129.8 -> 1140.5Improved
apply FastV (K=0, R=50%) to LLaVA-1.5-13BLLaVA-1.5-13B · A-OKVQA latencylatency per example: 0.539s -> 0.341sImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy