Research map / Vision-language and multimodal / Multimodal LLMs

Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives inference-compute scaling laws for VLMs trading off LLM size against visual token count, finds that visual reasoning favors the largest affordable LLM with very few (often one) visual tokens, and proposes a query-based convolutional cross-attention compressor (QueCC) for extreme compression.

Problem

Under a fixed inference FLOPs budget, it is unclear how to trade off the number of visual tokens passed to the LLM against the LLM's parameter count, and existing token compression work only studies modest compression ratios.

Approach

Fits a power-law scaling law for VLM error as a function of LLM parameters N and total inference tokens T, using projector-based token compression (TokenPacker variant) to vary visual tokens, and finds that for visual reasoning the optimal configuration is the largest LLM with very few visual tokens; it then adds user-query injection and learnable depthwise-convolution downsampling with local cros

Key results

• Fitted scaling law gives alpha=0.077 for LLM size versus beta=0.015 for visual tokens, so error grows roughly 5x faster with LLM parameter reduction than with token reduction for visual reasoning.
• Scaling law fitted on 0.5-7B VLMs predicts 14B VLM performance with less than 2% error.
• QueCC with 1 visual token reaches 67.3 VQAv2 and 1269.1 MME on the Vicuna-1.5 7B backbone, outperforming TokenPacker at 1 token (66.9 VQAv2, 1262.4 MME) on most benchmarks.

Measured effects reported in the paper

Surprising findings

• For visual reasoning, the compute-optimal configuration uses a single visual token with the largest affordable LLM, a regime far more aggressive than the 5-10x compression typical in token reduction literature.
• Repeating visual tokens to scale test-time compute yields no performance gains on the evaluated VLM tasks.

Limitations

• Scaling laws are fit on a single LLM family (Qwen-1.5) and a single compression projector, with generalization checked only partially (LLaVA-PruMerge).
• Extreme compression is not optimal for OCR and document tasks (DocVQA, TextVQA), where token count dominates performance.
• Analysis ignores vision encoder cost and generated tokens G, and focuses on short-answer tasks.

Experimental setup

Datasets: LLaVA-Next pretraining and finetuning data LLaVA-1.5 training data DocVQA TextVQA

Benchmarks: MME GQA AI2D MMBench MMMU ScienceQA MathVista POPE ChartQA DocVQA TextVQA VizWiz

Metrics: normalized average error across benchmarks accuracy F1 (POPE)

Baselines: TokenPacker LLaVA-PruMerge Matryoshka Multimodal Models Matryoshka Query Transformer LLaMA-VID LLaVA-1.5

Builds on: TokenPacker LLaVA-1.5 LLaVA-PruMerge Chinchilla scaling laws Matryoshka Query Transformer

Closest papers by meaning

• Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.93)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.93)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.92)
• ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.92)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.92)
• HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models (AAAI 2025 · similarity 0.92)