Research map / Large language models / Reasoning and chain of thought

TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TALON, a training-free, budget-driven adaptive draft-tree expansion for speculative decoding that uses top-K at the root and confidence-gated expansion below, improving speedup over EAGLE-3.

Problem

Existing tree-based speculative decoding builds fixed-width, fixed-depth draft trees that do not adapt to varying token difficulty, wasting draft compute on uncertain branches and truncating confident paths.

Approach

Grow the draft tree iteratively under a global node budget: use a fixed top-K expansion at the root for robustness to draft over-confidence, then gate deeper expansion by keeping candidates whose path probability is within a margin mu of the best candidate, yielding deep-and-narrow trees for easy contexts and shallow-and-wide trees for uncertain ones.

Key results

• On Vicuna-13B HumanEval (temperature 0), TALON reaches 5.16x speedup vs 4.77x for EAGLE-3.
• On Qwen3-8B GSM8K, TALON achieves 2.67x vs 2.37x for EAGLE-3 at temperature 0.
• TALON outperforms EAGLE-3 in all reported model-dataset settings at temperature 0 (speedups up to 5.16x over autoregressive decoding).

Surprising findings

• Removing the root top-K initialization (k=0) only slightly lowers speed (e.g., 263.8 to 261.5 tok/s on Llama3-8B), indicating the robust initialization gain is small.
• Extending the top-K initialization to deeper layers (k>=2) gives suboptimal returns, so one static layer is best.

Limitations

• Evaluated only at batch size 1; scalability to large-batch serving is untested.
• Hyperparameters mu and N are fixed globally and may need task-specific tuning for specialized domains.
• Confidence-gating threshold optimal values vary by model alignment (e.g., Qwen3-8B benefits from mu=0.01 while default is 0.03).

Experimental setup

Datasets: MT-Bench Alpaca GSM8K HumanEval CNN/DM QA

Benchmarks: MT-Bench Alpaca GSM8K HumanEval CNN/DM QA

Metrics: mean accepted tokens wall-time speedup draft efficiency

Baselines: EAGLE-3 SD Medusa Hydra OPT-Tree

Builds on: EAGLE-3 SpecInfer OPT-Tree Medusa

Closest papers by meaning

• PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.92)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.92)
• SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.92)
• SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.92)
• TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding (NeurIPS 2026 · similarity 0.92)
• SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.92)