AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

IVTP: Instruction-guided Visual Token Pruning for Large Vision-Language Models

Kai Huang, Hao Zou, Ye Xi, Bochen Wang, Zhen Xie, Liang Yu

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 17 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use attention rollout instead of plain current-layer CLS attention for token importance (ablation)IVTP GTP module · 12 benchmarks avgaverage accuracy (retrained): 58.1 -> 62.0Improved
prune per group of layers with rollout instead of layer-wise (ablation)IVTP · 12 benchmarks avgaverage accuracy (retrained): 59.5 -> 62.0Improved
aggregate intra-group attention with residual rollout rather than mean, max or multiply (ablation)IVTP · 12 benchmarks avgaverage accuracy (retrained): 60.7 (best alternative, max) -> 62.0Improved
add instruction-guided pseudo CLS token in LLM stage (ablation)IVTP · 12 benchmarks avgaverage accuracy (retrained): 59.2 -> 62.0Improved
use CLIP text encoder to pick key text tokens for the pseudo CLS (ablation)IVTP · 12 benchmarks avgaverage accuracy (retrained): 59.9 -> 62.0Improved
extend visual-only pruning methods (TopK, ToMe) into the LLM (ablation)TopK and ToMe · 12 benchmarks avgaverage accuracy (retrained): 60.0 -> 60.1 (ToMe)No clear effect
add LLM-stage pruning to ViT-only IVTP (ablation)IVTP-V · 12 benchmarks avgaverage accuracy (retrained): 60.8 -> 62.0Improved
group 3 layers per pruning group versus 2, 4, 6 or all (ablation)IVTP · 12 benchmarks avgaverage accuracy (retrained): 58.2 (ALL) -> 62.0 (3 layers)Improved
reduce visual tokens from 576 to 256-512 with IVTPLLaVA-1.5-7B · 12 benchmarks avgaverage accuracyNo clear effect
IVTP versus other pruning methods at 64 tokensLLaVA-1.5-7B · 12 benchmarks avgaverage accuracy: 60.0 (ToMe) -> 62.0Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy