Research map / Vision-language and multimodal / Multimodal LLMs

One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ALVTS, which uses a lightweight low-rank token selector per LLM layer to route only important visual tokens through each decoder layer while the rest skip it and are reintegrated later, so no token is permanently pruned.

Problem

Visual token pruning in LVLMs is static and irreversible: tokens pruned at one layer become inaccessible to all later layers, causing information loss that hurts accuracy, while LVLMs have very long visual token sequences that are costly to process.

Approach

Instead of a fixed pruned token subset, each decoder layer independently selects top-k visual tokens using importance scores from low-rank (SVD-initialized) approximations of the layer's Q/K projections, lets skipped tokens bypass that layer, and merges them back before the next layer. Only the selector parameters are optimized to match full-attention importance scores, with the LVLM frozen.

Key results

• At 89% token reduction on LLaVA-1.5-7B (64 tokens), ALVTS retains 96.73% average performance vs 93.27% for DART and 85.13% for FastV.
• On LLaVA-NeXT-7B with 320 of 2880 tokens, ALVTS retains 96.25% average performance, outperforming PyramidDrop and DART by 1.7% and 2.4%.
• On Qwen2.5-VL-3B at 89% reduction, ALVTS retains 86.39% vs 80.23% for FastV and 78.43% for PyramidDrop; prefill speedup of 1.6x on LLaVA-1.5-7B.

Surprising findings

• VTW collapses to near-zero on POPE and COCO for LLaVA-NeXT (e.g., POPE 0.27), showing that withdrawing visual tokens at one layer can be catastrophic.

Limitations

• Selector still computes low-rank Q/K projections over all visual tokens at every layer, so overhead grows with layers and token count.
• Evaluated only on a limited set of 8 benchmarks and three model families; no results on larger LVLMs beyond LLaVA-1.5-13B and Qwen2.5-VL-3B.

Experimental setup

Datasets: LLaVA-655k AI2D POPE TextVQA OKVQA VizWiz COCO Caption NoCaps RealWorldQA

Benchmarks: AI2D POPE TextVQA OKVQA VizWiz COCO Caption NoCaps RealWorldQA

Metrics: accuracy CIDEr average relative performance vs. vanilla model latency prefill time L2 distance of importance scores

Baselines: FastV VTW PyramidDrop DART

Builds on: FastV PyramidDrop DART VTW

Closest papers by meaning

• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.93)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.92)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model (CVPR 2025 · similarity 0.92)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.92)
• TransPrune: Token Transition Pruning for Efficient Large Vision-Language Model (CVPR 2026 · similarity 0.92)