Research map / Large language models / Reasoning and chain of thought

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SPEED-Bench, a benchmark for speculative decoding with a semantically diverse Qualitative split (880 prompts, 11 categories) and a Throughput split (fixed ISL buckets 1k-32k, batch sizes up to 512), plus a unified measurement framework over vLLM, TensorRT-LLM and SGLang.

Problem

Existing speculative decoding (SD) evaluations use low-diversity, short-prompt datasets and BS=1 settings on high-level libraries, so measured acceptance and speedups do not reflect production serving.

Approach

Curate a small, semantically diverse prompt set by greedy selection with local swap refinement minimizing pairwise embedding similarity, and pair it with a throughput split of fixed-ISL, large-batch workloads measured through production engines under a unified external tokenization pipeline.

Key results

• Synthetic random-token inputs overestimate SD throughput by an average of 23% relative to SPEED-Bench.
• On the Qualitative split at BS=32 and DL=3, EAGLE3 on Llama 3.3 70B reaches mean speedup 1.90x vs 1.60x for vanilla SD and 0.88x for N-gram at temperature 0.
• Greedy selection with swap refinement reduces average semantic similarity by 40% vs SpecBench (83% in Multilingual), and cuts Kendall-tau category-ranking disagreement at DL=7 by about 14x vs uniform sampling.

Surprising findings

• Random-token inputs also misestimate baseline autoregressive step latency for MoE models because they fail to trigger realistic expert routing, activating 20-30% fewer experts in some layers.
• Vocabulary pruning in EAGLE3 drafters degrades acceptance notably in Multilingual, RAG and Summarization, while Math and Coding are barely affected.

Limitations

• Client-side Python GIL overheads may distort measurements at BS>256.
• Evaluation focuses on draft-chain methods; tree-based experiments are only a supplementary demonstration on one model.
• Qualitative split is small, so per-category statistics retain some variance.

Experimental setup

Datasets: SpecBench Humanity's Last Exam LiveCodeBench Lite Code Contests HumanEvalPack RoleBench CoSER WritingBench Creative Writing V3 MT-Bench 101 MMLU-Pro MMATH

Benchmarks: SPEED-Bench Qualitative split SPEED-Bench Throughput split SpecBench

Metrics: acceptance length acceptance rate speedup output TPS user TPS TTFT

Baselines: SpecBench MT-Bench random token inputs N-Gram Vanilla SD

Builds on: EAGLE3 SpecBench Medusa vLLM TensorRT-LLM

Closest papers by meaning

• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.94)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.94)
• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.93)
• MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.93)
• Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.93)
• SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding (ICML 2026 · similarity 0.93)