Research map / Vision-language and multimodal / Multimodal LLMs

Efficient Large Multi-modal Models via Visual Context Compression

NeurIPS 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows visual tokens in LLaVA are redundant (average pooling inside the LLM drops 70% of tokens for ~3% GQA loss) and proposes LLaVolta, a staged training scheme that progressively compresses visual tokens during training to cut training time ~16-19% while matching or exceeding LLaVA-1.5-7B.

Problem

Visual tokens in multi-modal LLMs are numerous and redundant, but efficient training (not just inference) of MLLMs with compressed visual tokens has been largely unexplored; naive compression during training can lose information.

Approach

A nonparametric average-pooling compressor inserted at a deep LLM layer serves as the visual context compressor; LLaVolta trains in stages that progressively reduce compression (heavy to none/light), so the model is trained efficiently yet matches inference without compression.

Key results

• Average pooling of visual tokens at inference (stride 4 from layer 16) drops GQA by only ~3% while removing 70% of visual tokens in pre-trained LLaVA-1.5-7B.
• The three-stage 'last stage compression' LLaVolta scheme reaches 62.1% average over 13 benchmarks vs 61.8% for the LLaVA-1.5-7B baseline, cutting training time from 15.3h to 12.4h.
• On video (VideoLLaVA), the derived schemes reduce training time by about 9% with comparable or slightly higher accuracy.

Measured effects reported in the paper

Surprising findings

• Simple nonparametric average pooling outperforms attention-based (FastV, VCC) and parametric (Q-Former) compressors when used during training.
• Multi-stage layer-wise progressive compression underperforms single-stage after retraining, despite better direct inference.

Limitations

• Gains in training time are modest (roughly 16-19%) and video gains only about 9%, and the video setup lacks DeepSpeed ZeRO-3.
• Results are averaged over three runs only in some tables; the LLaVA-1.5 baseline was reproduced because some instruction-tuning image links were invalid.
• Stage-splitting and compression-layer choices are hand-designed and explored over a limited set of schemes.

Experimental setup

Datasets: LLaVA-1.5 instruction tuning data LLaVA-1.5 pretraining data Video-LLaVA training data

Benchmarks: GQA MM-Vet ScienceQA MME TextVQA POPE MMBench MMBench-CN VQA-v2 LLaVA-Bench-in-the-Wild VisWiz SEED-Image

Metrics: accuracy score latency (ms) TFLOPs training time (h) compression ratio

Baselines: LLaVA-1.5-7B FastV VCC K-Means Random Dropping Q-Former (MiniGPT-4) Mini-Gemini-2B LLaVA-13B Video-LLaVA

Builds on: LLaVA-1.5 FastV VCC BLIP-2 Q-Former Video-LLaVA

Closest papers by meaning

• Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information (AAAI 2025 · similarity 0.92)
• Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.92)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.92)
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.91)
• Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration (AAAI 2026 · similarity 0.90)
• EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.90)