Research map / Large language models / Reasoning and chain of thought

Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LTD, which trains two RL policies (depth and verification size) on top of Eagle3 to maximize per-cycle throughput of speculative decoding, reporting 2.24x-4.32x speedups.

Problem

Speculative decoding uses static or proxy-metric (acceptance length) driven draft-tree sizing, ignoring the real time cost of drafting and verification and treating the two phases in isolation.

Approach

Formulate draft-tree depth and verification size selection as an RL environment with two co-adaptive PPO policies whose reward is the wall-clock throughput (accepted tokens / cycle time), trained iteratively on top of Eagle3.

Key results

• LTD achieves mean greedy speedups of 3.92x on Llama-3.1-8B and 4.32x on Vicuna-13B, outperforming Eagle3 default by up to 36.4% (Qwen3-32B, 2.47x vs grid-searched 1.81x).
• At temperature 1, LTD keeps the highest average speedup on Vicuna-13B (3.62x vs 3.44x for Eagle3) while dynamic baselines degrade.
• Iterative co-adaptation improves Dpsk-8B mean speedup from 3.91x (Iter0) to 4.16x (Iter3).

Surprising findings

• LTD often has a shorter acceptance length than grid-searched Eagle3 yet achieves higher speedup, showing that acceptance length is a poor proxy for throughput.
• Using acceptance length as the size-policy reward yields lower speedup than throughput because excessive verification tokens inflate verification time.

Limitations

• Policies are trained only on HumanEval greedy decoding; generalization relies on evaluation across tasks.
• Gains over Grid-Search Eagle3 are modest for Llama and Dpsk (about 1-4%), and Qwen results are compared only against grid search.
• Method is tied to the Eagle3 tree structure and requires retraining policies per model.

Experimental setup

Datasets: HumanEval MT-bench GSM8K Alpaca Natural Questions MMLU

Benchmarks: MT-bench GSM8K Alpaca Natural Questions MMLU

Metrics: speedup ratio average acceptance length

Baselines: Eagle3 Eagle3+GS DDD SVIP Gammatune Disco SpecDec++ C2T

Builds on: Eagle3 PPO SpecInfer Spec-Bench

Closest papers by meaning

• Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.93)
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing (NeurIPS 2026 · similarity 0.93)
• Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.93)
• Steering Pretrained Drafters During Speculative Decoding (AAAI 2026 · similarity 0.92)
• Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding (ICLR 2026 · similarity 0.92)
• Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation (NeurIPS 2026 · similarity 0.92)