Research map / Vision-language and multimodal / Multimodal LLMs

HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HAWK, a training-free visual token pruning method that weights text-guided attention scores by per-head importance estimated once offline from head-ablation performance drops.

Problem

Visual tokens in MLLMs greatly increase inference cost; existing attention-based pruning methods assume all attention heads contribute equally to visual interpretation.

Approach

Estimate static head importance weights by ablating each attention head's visual access and measuring benchmark drops, then combine these weights with position-agnostic text-to-visual attention from the first LLM layer to score and prune visual tokens.

Key results

• On Qwen2.5-VL-7B with fixed resolution at 80.2% pruning (256 tokens), HAWK retains 96.0% of original performance vs 92.6% for the best baseline (FastV).
• On InternVL3-8B at 80% pruning, HAWK retains 94.1% vs 74.8% for CDPruner.
• On MME with Qwen2.5-VL-7B at 0.8 ratio, HAWK reduces E2E latency from 20m15s to 15m04s (x1.34 speedup) with score 2311 vs 2315 original.

Surprising findings

• Masking certain attention heads (e.g., Head 27) yields no degradation or slight gains on some benchmarks, suggesting some heads encode redundant information or noise.
• CDPruner performs notably worse than attention-only or similarity-only baselines on native-resolution and InternVL3 settings (e.g., 74.8% at 80% pruning on InternVL3-8B).

Limitations

• Head importance weights require one-time offline ablation over several benchmarks, and the authors have not released the head weights yet.
• Performance on some benchmarks (e.g., CCBench at 90% pruning) falls below the original model and is not uniformly best across all settings.

Experimental setup

Datasets: POPE MME HallBench ScienceQA-IMG RealWorldQA CCBench TextVQA OCRVQA ChartQA AI2D

Benchmarks: POPE HallBench MME ScienceQA-IMG RealWorldQA CCBench TextVQA OCRVQA ChartQA AI2D Video-MME WorldSense

Metrics: accuracy relative performance (Rel.) E2E latency GPU memory KV cache size

Baselines: FastV DivPrune CDPruner DART VisionZip

Builds on: FastV DivPrune CDPruner

Closest papers by meaning

• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.92)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.92)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.92)
• Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (NeurIPS 2025 · similarity 0.92)
• Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.92)