Research map / Vision-language and multimodal / Multimodal LLMs

LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LLaVA-Mini, which compresses each image or video frame to one vision token by fusing visual information into text tokens in a modality pre-fusion stage before the LLM, matching LLaVA-v1.5 on image benchmarks with 1 token instead of 576.

Problem

Large multimodal models incur high compute and latency because hundreds of vision tokens per image enter the LLM context, and prior token reduction methods that compress after the vision encoder lose visual information.

Approach

Layer-wise attention analysis shows vision tokens matter mainly in early LLM layers where text tokens fuse visual information. LLaVA-Mini adds a modality pre-fusion module (Transformer blocks copying the LLM structure) that fuses vision tokens into text tokens before the LLM, plus a query-based cross-attention compressor that reduces vision tokens to C x C (1 by default), so the LLM sees only one

Key results

• LLaVA-Mini with 1 vision token reaches 57.9% average on 11 image benchmarks vs 56.3% for LLaVA-v1.5 with 576 tokens, with 77% FLOPs reduction.
• LLaVA-Mini-HD (64 tokens, 672px) averages 58.6% and uses 8.13 TFLOPs vs 8.55 TFLOPs for LLaVA-v1.5.
• On MVBench LLaVA-Mini reaches 44.5% average, above Video-LLaVA (43.1%); on MLVU it reaches 42.8% average.

Measured effects reported in the paper

Surprising findings

• Removing all vision tokens in some later layers of LLaVA-v1.5 retains part of visual understanding ability, suggesting late-layer vision tokens are less critical.
• Without pre-fusion, compressing to 144 tokens still falls short of LLaVA-v1.5 by a notable margin, while pre-fusion with 1 token closes most of the gap.

Limitations

• Performance drops on some benchmarks (e.g., MME, POPE, VQAv2 slightly) relative to LLaVA-v1.5 with 576 tokens.
• Long-video results rely on length extrapolation since the model was trained only on short videos.
• Latency measured without engineering acceleration techniques.

Experimental setup

Datasets: LLaVA-558K captions LLaVA-665K instruction data Video-ChatGPT 100K video instruction data open-source instruction data

Benchmarks: VQAv2 GQA VisWiz ScienceQA-IMG TextVQA POPE MME MMBench SEED-Bench LLaVA-Bench-in-the-Wild MM-Vet MSVD-QA

Metrics: accuracy score (1-5) FLOPs latency GPU memory

Baselines: LLaVA-v1.5 MQT-LLaVA PruMerge PruMerge++ LLaMA-VID VoCo-LLaMA TokenPacker Video-LLaVA MovieChat Video-ChatGPT

Builds on: LLaVA-v1.5 Video-ChatGPT CLIP ViT-L/336px

Closest papers by meaning

• iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.94)
• LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models (ICLR 2025 · similarity 0.92)
• Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.91)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.90)
• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.90)
• Matryoshka Multimodal Models (ICLR 2025 · similarity 0.90)