Research map / Vision-language and multimodal / Multimodal LLMs

ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ViSpec, a speculative decoding framework for vision-language models that compresses image tokens with a vision adaptor, injects a global image feature into the draft model, and trains on synthetic long-response data, achieving up to 3.22x speedup.

Problem

Speculative decoding gives only modest speedups (<1.5x) on vision-language models because small draft models struggle to process long, redundant image token sequences while keeping text coherence.

Approach

A lightweight Q-Former-style vision adaptor compresses image tokens into a few compact tokens fed to a single-layer draft model with positional info preserved, a global image feature is added to all subsequent text hidden states, and training uses sampled target outputs plus multi-token prediction with synthetic long responses to avoid shortcut learning from target hidden states.

Key results

• ViSpec reaches up to 3.22x speedup (COCO Captions, LLaVA-1.6 7B, temperature 0), versus EAGLE-2 at up to 1.80x and Medusa at up to 1.61x on the same setting.
• Average speedup at temperature 0 is 2.58x for LLaVA-1.6 7B, 2.38x for LLaVA-1.6 13B, 1.87x for Qwen2.5-VL 3B, and 1.80x for Qwen2.5-VL 7B.
• On high-resolution HR-Bench 4K, ViSpec gives 1.93x speedup on LLaVA-1.6 7B versus 1.52x for EAGLE-2, and 1.29x versus 0.90x on Qwen2.5-VL 7B.

Measured effects reported in the paper

Surprising findings

• EAGLE-2 falls below 1x speedup on some Qwen2.5-VL settings (e.g., 0.83x on MVBench and 0.90x on HR-Bench 4K), indicating slowdowns from prefill cost on high-resolution inputs.
• Increasing the number of compressed image embeddings beyond one mostly did not improve acceptance length and reduced speedup.

Limitations

• Absolute speedups (about 1.8x to 2.6x average) trail state-of-the-art text-only speculative decoding, as the authors acknowledge.
• Qwen2.5-VL speedups are lower than LLaVA-1.6 speedups, and high-resolution inputs lengthen prefill, reducing end-to-end gains.
• Video results come from a draft model trained only on static images and are preliminary.

Experimental setup

Datasets: ShareGPT LLaVA Visual Instruct Pretrain LCS HR-Bench 4K MME-RealWorld MSVD-QA MVBench

Benchmarks: ScienceQA MM-Vet TextVQA MME COCO Captions VizWiz GQA SEED-Bench HR-Bench 4K MME-RealWorld MSVD-QA MVBench

Metrics: average acceptance length speedup ratio

Baselines: Medusa EAGLE-2

Builds on: EAGLE-2 EAGLE Medusa Q-Former speculative decoding

Closest papers by meaning

• SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models (NeurIPS 2026 · similarity 0.91)
• Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.91)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.90)
• Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.89)
• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.89)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.89)