Research map / Large language models / Reasoning and chain of thought

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VIA-SD, a three-tier speculative decoding scheme where a layer-skipped slim verifier derived from the full model (chosen via DIMR routing search) handles medium-confidence draft tokens before escalation to the full verifier.

Problem

Standard speculative decoding uses a binary accept-or-recompute decision, so many draft tokens rejected by the drafter still force expensive full-verifier calls even though a cheaper model could verify them.

Approach

Introduce an intermediate verifier built as a routed intra-model submodel of the full verifier (layers skipped via a mask chosen by dynamic intra-model routing with random search and Bayesian optimization), giving a three-tier draft/slim-verify/full-verify hierarchy justified by a KL information-projection argument.

Key results

• On Gemma2 2B→27B WebQuestions, VIA-SD reduces rejection rate from 0.27 (speculative decoding) to 0.14 and reaches 2.32× speedup, versus 1.55× for speculative decoding.
• On WMT14 En-De with T5 S→XL, VIA-SD reaches 3.35× speedup with 0.21 rejection rate versus 1.80× for speculative decoding.
• Reported gains are 10–20% speedup over strong SD baselines (Faster Cascades), with rejection rates reduced by 0.10–0.22 across tasks.

Surprising findings

• Three-tier verification sometimes slightly exceeds direct target-model decoding quality, which the authors attribute to lossy gating reshaping the effective distribution.
• Random skip-layer routing gives marginal speedup and lower accuracy than the DIMR-selected routed verifier, and an independent 13B intermediate model gives higher speed but at 1.38× peak memory.

Limitations

• Speedups are measured only on A100 GPUs under controlled settings and may vary with batch size, hardware, and KV-cache implementation.
• Offline DIMR search costs 0.30–1.13 GPU-hours per model pair, and the slim-verifier structure is fixed after search.
• Speedup gains are modest against strong cascade baselines (around 10–20%) and the paper's theoretical KL decomposition is an inequality-based argument with approximations in practice.

Experimental setup

Datasets: XSum CNN/DailyMail WMT14 En-De GSM8K MBPP SQuAD 2.0 WebQuestions NaturalQA TriviaQA

Benchmarks: XSum CNN/DailyMail WMT14 En-De WebQuestions NaturalQA TriviaQA GSM8K MBPP SQuAD 2.0

Metrics: rejection rate speedup accuracy ROUGE-2 BLEU

Baselines: Speculative Decoding Cascade SD Faster Cascades EAGLE PEARL

Builds on: speculative decoding lossy speculative decoding cascade speculative drafting Faster Cascades EAGLE PEARL

Closest papers by meaning

• TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification (NeurIPS 2026 · similarity 0.92)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.91)
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.91)
• Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.91)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.91)