Research map / Vision-language and multimodal / Multimodal LLMs

SP$^2$ec: Adaptive Self-Speculative Decoding for Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SP2EC, a training-free single-peak bandit that adapts the draft configuration (retained layers or visual tokens) per prompt for self-speculative decoding in VLMs, achieving up to ~11% gains over base self-speculative methods.

Problem

Self-speculative decoding for VLMs needs a per-prompt choice of draft configuration (retained layers or visual tokens), and the optimal choice varies across prompts; exploring configurations online costs wall time.

Approach

Order draft configurations into a sequence where acceptance shows diminishing returns and time is affine in size, so throughput is single-peaked; a bandit then explores only the leader's neighborhood and minimizes wall-time regret rather than treating arms independently.

Key results

• On Qwen2.5-VL-7B-Instruct, KnapSpec + SP2EC reaches 1.35x average speedup versus 1.29x for KnapSpec alone, while KnapSpec + UCBSpec drops to 1.24x.
• SpecVLM + SP2EC achieves 1.56x average speedup on Qwen2.5-VL-7B-Instruct versus 1.47x for SpecVLM.
• SP2EC confines only 6.1-16.1% of arm pulls outside the top three arms, compared with 49.1-53.1% for UCBSpec.

Surprising findings

• Higher mean accepted tokens do not necessarily yield higher speedup, because more accurate drafters incur greater drafting cost.
• Standard UCB-style exploration (UCBSpec) can reduce speedup relative to the non-adaptive baseline because exploratory pulls on slow configurations cost wall time.

Limitations

• Theory relies on structural assumptions (diminishing returns, affine cost) on the ordered candidate path.
• Experiments cover only two VLM families (Qwen2.5-VL and LLaVA-NeXT) plus limited extra models.
• Gains over base methods are modest (roughly 0.06x-0.09x average).

Experimental setup

Datasets: Video-MME MMBench-Video ARKitScenes AIME25

Benchmarks: Video-MME MMBench-Video ARKitScenes AIME25

Metrics: mean accepted tokens speedup ratio wall-time regret tokens per second

Baselines: UCBSpec SWIFT KnapSpec SpecVLM EAGLE3 COSSIM

Builds on: BanditSpec SWIFT KnapSpec SpecVLM UCB

Closest papers by meaning

• ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding (NeurIPS 2025 · similarity 0.91)
• SEED: Self-Speculative Decoding via Implicit Encoder–Decoder (NeurIPS 2026 · similarity 0.89)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.89)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.89)
• BanditSpec: Adaptive Speculative Decoding via Bandit Algorithms (ICML 2025 · similarity 0.88)
• Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.88)