Research map / Vision-language and multimodal / Multimodal LLMs

Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Dynamic-LLaVA, which learns lightweight predictors to sparsify both image tokens (prefill) and generated output tokens (decoding, with and without KV cache) in LLaVA-style MLLMs, trained end-to-end with masked softmax and Gumbel-Softmax.

Problem

Existing MLLM token reduction methods only cut image tokens, so their efficiency gains in prefill shrink as decoding generates more output tokens, and KV cache memory keeps growing during generation.

Approach

Two learnable predictors decide which image tokens (prefill) and which generated output text tokens (decoding) to keep, with tailored sparsification per inference mode; training uses masked softmax to preserve autoregressive loss computation and Gumbel-Softmax with straight-through estimation for end-to-end differentiable masks.

Key results

• Dynamic-LLaVA-7B (vision-only sparsification) keeps 115 of 576 image tokens (-80%) and reaches 78.0 on VQAv2 vs 78.5 for LLaVA-1.5-7B, with about 75% prefill TFLOPs reduction.
• With vision and language sparsification, decoding without KV cache uses about 45-50% fewer TFLOPs and decoding with KV cache about 39-50% less memory on LVIS-VQA, with PPL rising by less than 0.3.
• On LLaVA-13B, decoding with KV cache at 4K generated tokens drops peak memory from OOM (LLaVA-1.5-13B) to 56G for Dynamic-LLaVA-13B, while the batch-8 generation time is about 2.2x faster at 4K tokens.

Measured effects reported in the paper

Surprising findings

• Sparsifying output text tokens did not hurt, and sometimes slightly improved, vision benchmark scores (e.g., MMVP 31.3 vs 29.3 for 13B).
• Sparsifying text tokens during prefill (30%) degraded performance by about 1.45% on average, so prefill text tokens are important.

Limitations

• Output text sparsification is only applied to samples whose output exceeds a minimum length, due to training instability on short outputs.
• Text-token sparsification during prefill causes noticeable degradation.
• Requires additional end-to-end training of the MLLM and predictors.

Experimental setup

Datasets: 656K Mixture Dataset (LLaVA-1.5) LVIS-Instruct4V ShareGPT4V

Benchmarks: VQAv2 GQA SciQA TextVQA POPE MME MMBench SEED MMVP RealWorldQA CVBench-2D MM-Vet

Metrics: accuracy PPL METEOR TFLOPs GPU memory latency

Baselines: LLaVA-1.5 FastV LLaVA-PruMerge+ VoCo-LLaMA LLaVA-HiRED H2O TokenPacker SparseVLM TRIM IVTP Random token dropping Structure token dropping

Builds on: LLaVA-1.5 FastV H2O TokenPacker Gumbel-Softmax

Closest papers by meaning

• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.92)
• Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.91)
• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.91)
• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.91)
• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.91)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)