Research map / Vision-language and multimodal / Multimodal LLMs

Sharp Eyes and Memory for VideoLLMs: Information-Aware Visual Token Pruning for Efficient and Reliable VideoLLM Reasoning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SharpV, a training-free two-stage pruning framework for VideoLLMs that adaptively prunes visual tokens pre-LLM using spatio-temporal dissimilarity scores and prunes KV cache intra-LLM via visual-token similarity to original features, without needing attention scores.

Problem

VideoLLMs incur quadratic compute and KV-cache scaling due to many redundant visual tokens; existing pruning uses fixed ratios and attention-score-based methods that are incompatible with Flash Attention.

Approach

Pre-LLM: token importance from a Euclidean-on-unit-vectors dissimilarity combining temporal (adjacent-frame) and spatial (frame-mean) terms, with per-frame retention ratios set from frame-level motion norms, avoiding fixed ratios. Intra-LLM: discard a layer's visual KV cache when cosine similarity between its hidden visual states and the original visual features drops below a threshold, requiring

Key results

• On LLaVA-OneVision-7B, SharpV (Adaptive) at a 12% token budget reaches 62.2 average accuracy versus 61.9 for the dense model, with TTFT 0.64s (1.64x speedup) and 22G memory versus 32G.
• SharpV (Adaptive) retains about 32% of visual tokens pre-LLM on average and 39% of KV cache, giving the 12% overall budget.
• On LLaVA-OneVision-0.5B, SharpV (Adaptive) averages 49.8 versus 49.5 dense at a 12% budget, outperforming FastV, PruMerge and DyCoke at similar or lower budgets.

Surprising findings

• Adaptive pruning occasionally exceeds the dense model's accuracy (e.g., 62.2 vs 61.9 average on LLaVA-OV-7B), which the authors attribute to reduced visual noise.
• Random token selection under the same adaptive per-frame budgets achieves competitive accuracy, with V-Random-dagger close to Visual SharpV on several benchmarks.

Limitations

• Fine-detail sensitive scenes (e.g., micro-expressions) may suffer information loss.
• No rigorous theoretical framework for deep-layer token transformation; the degradation claim is empirical.
• Adaptive thresholds and hyperparameters (M, w, K) require tuning on ablations.

Experimental setup

Datasets: MVBench VideoMME NExT-QA ActivityNet-QA

Benchmarks: MVBench VideoMME NExT-QA ActivityNet-QA

Metrics: accuracy TTFT TPOT GPU memory TFlops visual retention ratio memory retention ratio

Baselines: FastV PruMerge DyCoke

Builds on: FastV DyCoke PruMerge FrameFusion

Closest papers by meaning

• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.90)
• FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.90)
• V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.90)
• ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning (AAAI 2026 · similarity 0.89)
• An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models (ICLR 2026 · similarity 0.89)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.89)