Research map / Large language models / Reasoning and chain of thought

Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PPOW, a window-level RL framework that trains EAGLE-family speculative drafters with a cost-aware speedup reward, a distribution-based proximity reward, and adaptive divergence-aware window sampling, reporting higher acceptance lengths and speedups than EAGLE-3 and GRIFFIN.

Problem

Speculative decoding drafters are trained with token-level supervised objectives, which do not directly optimize window-level, prefix-sensitive acceptance length, so hard-to-draft positions truncate speculative windows and limit speedup.

Approach

Formulate drafter training as group-relative policy optimization over fixed-length speculative windows, using a cost-aware reward based on accepted prefix length, an auxiliary reward for near-target windows when k=0, and sampling windows by confidence-weighted draft-target KL (criticality score).

Key results

• Across LLaMA-3.1-8B, LLaMA-3.3-70B, Qwen3-8B, and Qwen3-32B at temperature 0.0, PPOW reports mean acceptance lengths of 6.40, 6.29, 6.52, and 6.44 with speedups of 3.46x, 4.36x, 3.39x, and 3.92x.
• On LLaMA-3.1-8B GSM8K, PPOW reaches tau=6.50 vs 6.12 for EAGLE-3 and 5.72 for continued supervised training at 1e-5 LR over 30k steps.
• Ablation on LLaMA-3.1-8B: removing both R_dist and ADAW reduces MT-Bench tau from 5.47 to 4.38.

Surprising findings

• Continued supervised training initially improves acceptance length but later degrades, while PPOW keeps improving.
• The measured-speedup reward gave only a marginal gain over the cost-aware reward (6.61 vs 6.50 tau), suggesting the proxy reward is adequate.

Limitations

• Training requires grouped rollouts, verification and a frozen target model, making it more resource-intensive than supervised training.
• Several extra hyperparameters (window size K, gamma, eta, epsilon, curriculum) need tuning.
• Gains are smaller on open-ended tasks such as MT-Bench, X-SUM and WMT14.

Experimental setup

Datasets: ShareGPT UltraChat-200k

Benchmarks: MT-Bench HumanEval GSM8K X-SUM WMT14

Metrics: average acceptance length speedup ratio

Baselines: EAGLE-3 GRIFFIN OSD Lookahead FastDraft continued supervised training

Builds on: EAGLE-3 GRPO speculative decoding GRIFFIN

Closest papers by meaning

• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.93)
• SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting (NeurIPS 2026 · similarity 0.91)
• A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
• LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding (ICML 2026 · similarity 0.91)
• Block Verification Accelerates Speculative Decoding (ICLR 2025 · similarity 0.91)
• PEARL: Parallel Speculative Decoding with Adaptive Draft Length (ICLR 2025 · similarity 0.91)