Research map / Large language models / Reasoning and chain of thought

Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpecFormer, a lossless speculative-decoding draft module combining context causal attention with a bidirectional draft attention and position-specific FFN, aiming to keep speedups under large batch sizes with small draft-token budgets.

Problem

Speculative decoding relies on large draft trees, but under large-batch serving the idle compute that SD exploits shrinks, so draft token budgets become small and position-dependent draft parameters scale poorly.

Approach

Combine a causal attention over multi-level hidden states (HS[0], HS[L/2], HS[L-1], HS[L]) that reads the full context with a non-autoregressive bidirectional attention over draft positions, using a single position-specific linear projection (Positional FFN) to inject position information, so draft parameters are mostly position-independent.

Key results

• At batch size 1 with k=6 draft tokens, SpecFormer reaches 1.81x speedup (74 TPS vs 41 W/o SD) and kappa 2.22, versus EAGLE-3 at 1.75x and HASS at 1.74x.
• At batch size 128 with k=6, SpecFormer keeps 1.81x speedup (9308 vs 5143 TPS), while the baselines remain around 1.73x-1.74x.
• On Qwen3-4B with k=8 and batch size 1, SpecFormer reaches 1.54x speedup; speedup falls to 1.47x on Qwen3-14B, and self-distillation is needed (no-self-distill gives 0.94x).

Surprising findings

• Without self-distillation, the draft model gives negligible acceleration (0.94x at batch size 1), so alignment with the base model's own outputs is critical.
• Larger base models show a more favorable kappa-to-TPS conversion, though absolute speedup decreases with model size.

Limitations

• Speedup is shown mainly against a constrained draft-token budget, and baselines are not run at their native settings.
• Speedup decreases as base model size grows (1.56x at 4B to 1.47x at 14B).
• Bidirectional attention gives only a small gain in the ablation (kappa 1.81 vs 1.80 without it).

Experimental setup

Datasets: UltraChat-200K MT-Bench HumanEval GSM8K Alpaca CNN/DM

Benchmarks: MT-Bench HumanEval GSM8K Alpaca CNN/DM UltraChat-200K test split

Metrics: kappa (accepted length times draft length over draft tokens) tokens per second (TPS) speedup ratio average accepted length

Baselines: HASS EAGLE-3 Medusa W/o SD

Builds on: Medusa EAGLE HASS EAGLE-3 Multi-Token Prediction BiLD

Closest papers by meaning

• SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices (NeurIPS 2024 · similarity 0.93)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
• Speculative Speculative Decoding (ICLR 2026 · similarity 0.92)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.92)
• Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.92)