Research map / Vision-language and multimodal / Multimodal LLMs
HiDivDrop: Vision Token Reduction in MLLMs via Late Injection and Differentiable Top-K
Hao Wu, Yingqi Fan, Dai Jinyang, Junlong Tong, Yunpu Ma, Xiaoyu Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiDivDrop, a vision token reduction method for MLLMs that uses late injection and differentiable top-K selection to progressively prune vision tokens.
Problem
Progressive vision token pruning in MLLMs is limited: it treats shallow layers as crucial for fusion and uses suboptimal pruning, leaving computational savings unrealized, given the quadratic cost of vision tokens.
Approach
Injects vision tokens late rather than treating shallow layers as essential for fusion, and uses a differentiable top-K selection to progressively drop vision tokens during processing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use differentiable top-k instead of hard top-k for token selection (ablation) | HiDrop progressive pruning · 7 benchmarks (PT+FT) | avg % of baseline: 97.7 -> 99.7 | Improved |
| use differentiable top-k with finetune-only training (ablation) | HiDrop progressive pruning · 7 benchmarks (FT) | avg % of baseline: 97.5 -> 98.1 | Improved |
| use persistent position encoding instead of compacted (PDrop-style) (ablation) | HiDrop · 7 benchmarks | avg % of baseline: 96.4 -> 97.6 | Improved |
| use persistent PE instead of group PE (ablation) | HiDrop · 7 benchmarks | avg % of baseline: 97.0 -> 97.6 | Improved |
| estimate saliency with multi-round last-token attention vs single-round (ablation) | HiDrop · 7 benchmarks | avg % of baseline: 97.1 -> 99.7 | Improved |
| aggregate attention from all tokens with L2 norm instead of last token n-rounds (ablation) | HiDrop · 7 benchmarks | avg % of baseline: 99.7 -> 99.6 | No clear effect |
| choose ILVAS-based filtering layers {10,14,16,18} instead of control {12,15,18,21} (ablation) | HiDrop · 11 benchmarks avg | avg % of baseline: control lower than 98.3 | Improved |
| inject vision tokens at layer 9 instead of earlier or later (ablation) | HiDrop late injection · benchmark avg | avg % of baseline | Improved |
| add late injection, concave pyramid pruning and early exit (HiDrop) vs PDrop at 88.9% pruning | LLaVA-1.5-7B · 11 benchmarks avg | avg % of baseline: 94.2 -> 98.3 | Improved |
| add HiDrop pruning to LLaVA-1.5-7B | LLaVA-1.5-7B · training cost | GPU hours / TFLOPs: 159.3 -> 94.4 hours; 3.82T -> 0.42T | Improved |
| remove vision tokens in shallow/middle layers (early exit before fusion done) (ablation) | LLaVA-1.5-7B (training-free) · GQA | accuracy | Hurt |
| increase instruction data from 665k to 1M with HiDrop at 48 tokens (ablation) | HiDrop · 7 benchmarks | avg % of baseline: drop 3.0% (665k) vs 3.7% (1M) | No clear effect |
Surprising findings
- Shallow layers are not necessarily crucial for multimodal fusion, contrary to the assumption underlying progressive pruning schemes.
Experimental setup
Builds on: progressive vision token pruning
Closest papers by meaning
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.92)
- TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
- VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.91)
- ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models (CVPR 2026 · similarity 0.91)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.91)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
