AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

HiDivDrop: Vision Token Reduction in MLLMs via Late Injection and Differentiable Top-K

Hao Wu, Yingqi Fan, Dai Jinyang, Junlong Tong, Yunpu Ma, Xiaoyu Shen

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HiDivDrop, a vision token reduction method for MLLMs that uses late injection and differentiable top-K selection to progressively prune vision tokens.

Problem

Progressive vision token pruning in MLLMs is limited: it treats shallow layers as crucial for fusion and uses suboptimal pruning, leaving computational savings unrealized, given the quadratic cost of vision tokens.

Approach

Injects vision tokens late rather than treating shallow layers as essential for fusion, and uses a differentiable top-K selection to progressively drop vision tokens during processing.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use differentiable top-k instead of hard top-k for token selection (ablation)HiDrop progressive pruning · 7 benchmarks (PT+FT)avg % of baseline: 97.7 -> 99.7Improved
use differentiable top-k with finetune-only training (ablation)HiDrop progressive pruning · 7 benchmarks (FT)avg % of baseline: 97.5 -> 98.1Improved
use persistent position encoding instead of compacted (PDrop-style) (ablation)HiDrop · 7 benchmarksavg % of baseline: 96.4 -> 97.6Improved
use persistent PE instead of group PE (ablation)HiDrop · 7 benchmarksavg % of baseline: 97.0 -> 97.6Improved
estimate saliency with multi-round last-token attention vs single-round (ablation)HiDrop · 7 benchmarksavg % of baseline: 97.1 -> 99.7Improved
aggregate attention from all tokens with L2 norm instead of last token n-rounds (ablation)HiDrop · 7 benchmarksavg % of baseline: 99.7 -> 99.6No clear effect
choose ILVAS-based filtering layers {10,14,16,18} instead of control {12,15,18,21} (ablation)HiDrop · 11 benchmarks avgavg % of baseline: control lower than 98.3Improved
inject vision tokens at layer 9 instead of earlier or later (ablation)HiDrop late injection · benchmark avgavg % of baselineImproved
add late injection, concave pyramid pruning and early exit (HiDrop) vs PDrop at 88.9% pruningLLaVA-1.5-7B · 11 benchmarks avgavg % of baseline: 94.2 -> 98.3Improved
add HiDrop pruning to LLaVA-1.5-7BLLaVA-1.5-7B · training costGPU hours / TFLOPs: 159.3 -> 94.4 hours; 3.82T -> 0.42TImproved
remove vision tokens in shallow/middle layers (early exit before fusion done) (ablation)LLaVA-1.5-7B (training-free) · GQAaccuracyHurt
increase instruction data from 665k to 1M with HiDrop at 48 tokens (ablation)HiDrop · 7 benchmarksavg % of baseline: drop 3.0% (665k) vs 3.7% (1M)No clear effect

Surprising findings

Experimental setup

Builds on: progressive vision token pruning

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy