Research map / Vision-language and multimodal / Multimodal LLMs

Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes compressing visual tokens before the LLM by training a lightweight 5-layer 1D conv network to predict GAE-style explainability relevance from first-layer attention, enabling instruction-related pruning at LLM input with no MLLM changes.

Problem

MLLMs process many visual tokens, making prefill and KV-cache costs high; existing instruction-related token pruning happens inside intermediate LLM layers, and the question is whether task-related pruning can be done before the LLM.

Approach

Use generic attention explainability (GAE-style gradient-weighted relevance propagation) to score visual token importance w.r.t. the instruction, then learn a small fully convolutional mapper from first-layer attention (text-to-visual) to these relevance scores, so pruning happens at the LLM input without modifying the MLLM.

Key results

• On LLaVA-OneVision at 50% retention, the learned compressor reaches 97.4% average image preservation (vs 95.0% for DART) and 100.4% average video preservation at 50%.
• On Qwen2-VL at 25% retention, the compressor keeps 92.9% image and 94.2% video average performance, with prefilling speedup 1.60x and KV cache reduced from 71.2MB to 17.8MB on MMStar.
• The trained compressor (about 10K samples, 5-layer conv, roughly 0.24x FLOPs) outperforms FastV, PDrop and DART at comparable FLOPs across most image and video benchmarks.

Surprising findings

• Pruning visual tokens before the LLM (input stage) with explainability-guided relevance retains nearly full performance, contrary to the common assumption that compression must happen in intermediate LLM layers.
• Learning to predict relevance from only the first-layer attention map performs nearly as well as using multi-layer attention inputs, which add substantial FLOPs.

Limitations

• Relevance scores require backward passes and eager attention, making label generation resource-intensive and hard to scale to high-resolution images or long videos.
• A separate compressor must be trained for each MLLM.
• Compression gap from the explainability-based oracle to the learned compressor is more pronounced on image tasks.

Experimental setup

Datasets: LLaVA-Video Infinity-MM NExT-QA ActivityNetQA

Benchmarks: MME MMStar MMVet SEED-Bench POPE TextVQA MMBench Video-MME MVBench MMBench-Video NExT-QA ActivityNetQA

Metrics: accuracy performance preservation ratio FLOPs prefilling time KV cache memory total inference time

Baselines: FastV PyramidDrop DART PruneVID FastVID VisionZip

Builds on: GAE (Generic Attention Explainability) FastV PyramidDrop DART

Closest papers by meaning

• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.92)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.92)
• What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models (CVPR 2026 · similarity 0.92)
• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.92)
• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.91)