Research map / Vision-language and multimodal / Multimodal LLMs

iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes iLLaVA, a training-free token merging method that reduces visual tokens in both the image encoder and the LLM of LVLMs, recycling information from discarded tokens into cluster tokens, achieving ~2x throughput and ~4x prefilling speedup.

Problem

LVLM inference is expensive, and prior token reduction methods only prune image tokens inside or before the LLM, ignoring the image encoder, which is both a major compute cost and the main source of LLM input tokens.

Approach

Jointly merge visual tokens inside the image encoder (between attention and FFN in several blocks) and inside the LLM, keeping top-attention informative tokens plus recycled cluster tokens that aggregate information from discarded tokens via attention-weighted merging of their M nearest neighbors.

Key results

• At 88.9% image token reduction on Qwen2.5-VL 7B, iLLaVA retains 95.2% of vanilla average performance (62.8 vs 65.3), outperforming VisionZip (61.4) and other baselines.
• On MMMU, iLLaVA reduces memory by 1.59x, improves throughput by 2.12x and reduces prefilling time by 4.46x relative to vanilla.
• On video benchmarks at 95% token reduction, iLLaVA achieves 96.8% of vanilla average, 1.7% above VisionZip.

Surprising findings

• Token reduction inside the image encoder yields 25.3% higher throughput and 21.2% lower memory than reducing the same number of tokens inside the LLM.
• Pruning tokens in the image encoder early causes substantial performance degradation, motivating merging rather than pruning.

Limitations

• Performance drops more noticeably on document and chart benchmarks (DocVQA, InfoVQA, ChartQA) that need many image tokens.
• Small objects may be missed during token reduction, causing degradation versus the vanilla model.

Experimental setup

Datasets: MMMU MMMU Pro MMBench MMBench V1.1 MME MMStar MMVet MuirBench RealWorldQA VideoMME MVBench EgoSchema

Benchmarks: MMMU MMMU Pro MMBench MMBench V1.1 MME MMStar MMVet MuirBench RealWorldQA VideoMME MVBench EgoSchema

Metrics: accuracy (percent of vanilla) throughput prefilling time memory usage

Baselines: FasterVLM PyramidDrop SparseVLM VisionZip FastV AdaFV DivPrune FEATHER AIM LLaVA-PruMerge ToMe

Builds on: ToMe VisionZip FastV FlashAttention

Closest papers by meaning

• LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.94)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.91)
• FastVLM: Efficient Vision Encoding for Vision Language Models (CVPR 2025 · similarity 0.91)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.91)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.91)
• DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs (NeurIPS 2024 · similarity 0.91)