Research map / Large language models / Reasoning and chain of thought

SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpecBlock, a block-iterative tree drafter producing K dependent positions per forward with a layer-wise shift, rank-head-guided branching, valid-prefix loss masking, and cost-aware bandit serving-time adaptation for speculative decoding.

Problem

Speculative decoding drafters trade off path coherence against drafting cost: autoregressive drafters (EAGLE-3) preserve dependence but need one drafter call per tree depth, while parallel drafters are cheap but produce incoherent draft paths.

Approach

Each drafter forward produces a K-token block with dependence enforced by a per-layer shift of the previous position's hidden state; the tree grows by re-invoking the drafter from positions of earlier blocks; a co-trained rank head allocates per-position branching; a valid-prefix mask stops loss after the first wrong prediction; and a cost-aware bandit selectively updates the drafter at serving ti

Key results

• SpecBlock improves mean speedup over EAGLE-3 by 8-13% at 44-52% of its drafting cost (e.g., Llama-3.1-8B T=0 mean Spd 3.21x vs 2.92x, drafting share 16% vs 31%).
• Cost-aware adaptation (SpecBlock+adapt) extends the lead over EAGLE-3 to 11-19% on benchmarks with sufficient streaming queries.
• Layer-wise shift removal drops Llama-3.1-8B mean speedup from 3.21x to 2.95x and accepted length from 4.41 to 4.02.

Surprising findings

• Removing the rank-guided branching in the ablation leaves mean accepted length slightly higher (4.25 vs 4.41 is lower, but the head-only action ablation gives tau 4.58 above the full adapt tau 4.54), showing a speedup-vs-length trade-off in adaptation.

Limitations

• Adaptation is not evaluated on short benchmarks (HumanEval, MT-Bench) because traffic is too small to amortize update cost.
• Gains are measured under HuggingFace Transformers at batch size 1 on A100 GPUs only.
• Rank head classification is weak on middle buckets (F1 about 0.46-0.50).

Experimental setup

Datasets: UltraChat-200K ShareGPT MT-Bench HumanEval MATH-500 Alpaca Natural Questions WMT-23

Benchmarks: MT-Bench HumanEval MATH-500 Alpaca Natural Questions WMT-23

Metrics: speedup over vanilla decoding tokens per second accepted length drafting cost share

Baselines: SpS Medusa ParallelSpec Falcon EAGLE-3 SpecBlock+OSD

Builds on: EAGLE-3 speculative sampling Medusa ParallelSpec Online Speculative Decoding

Closest papers by meaning

• D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting (NeurIPS 2026 · similarity 0.93)
• BASTION: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Drafting (NeurIPS 2026 · similarity 0.93)
• TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees (NeurIPS 2026 · similarity 0.92)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
• TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding (NeurIPS 2026 · similarity 0.92)
• Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.92)