Research map / Large language models / Reasoning and chain of thought
Reward-Guided Speculative Decoding for Efficient LLM Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reward-Guided Speculative Decoding (RSD), which uses a process reward model to accept high-reward draft reasoning steps from a small model and falls back to a large target model otherwise, trading a controlled bias for efficiency on reasoning benchmarks.
Problem
Speculative decoding enforces strict unbiasedness, so it rejects correct draft tokens whose probability under the large model is low, wasting computation and limiting speedups on multi-step reasoning tasks.
Approach
Replace the exact-match acceptance test of speculative decoding with a step-level process reward threshold: draft reasoning steps with reward at or above delta are accepted, and the rest are regenerated by the target model, yielding a biased mixture distribution that can match or exceed the target model's accuracy at lower FLOPs.
Key results
- RSD (7B/72B/7B) reaches 88.0 on MATH500 vs 85.6 for the 72B target alone, with up to 4.4x fewer FLOPs.
- With Qwen2.5-Math 7B target and 1.5B draft, RSD averages 66.1 vs 65.3 for the single 7B model and 64.6 for SD.
- On AlpacaEval, RSD (Llama-3.2-1B/3.1-8B with ORM) reaches 18.85% win rate vs 7.09% for the draft alone and 24.47% for the target alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reward-guided decoding instead of the target model alone | Qwen2.5-Math-Instruct-7B target with 1.5B draft and 7B PRM · MATH500 | accuracy: 83.2 -> 84.6 | Helped |
| Use reward-guided decoding instead of speculative decoding | Qwen2.5-Math-Instruct-7B target with 1.5B draft and 7B PRM · GPQA Diamond | accuracy: 28.8 -> 33.8 | Helped |
| Use the optimized threshold δ* instead of fixed δ=0.7 (ablation) | RSD, Qwen2.5-Math-Instruct-7B target and 1.5B draft, 7B PRM · Average across six reasoning benchmarks | accuracy: 66.1~(+0.8) -> 66.5~(+1.2) | Helped |
| Increase PRM size from 1.5B to 7B (ablation) | RSD with Qwen2.5-Math-Instruct-7B target and 1.5B draft · MATH500 | accuracy: 82.6 -> 84.6 | Helped |
| Replace Skywork-o1-Open-PRM-7B with Qwen2.5-Math-PRM-7B (ablation) | RSD with Qwen2.5-Math-Instruct-7B target and 1.5B draft · MATH500 | accuracy: 84.6 -> 83.2 | Hurt |
| Use Qwen2.5-Math-PRM-72B instead of Skywork-o1-Open-PRM-7B (ablation) | RSD with Qwen2.5-Math-Instruct-7B target and 1.5B draft · MATH500 | accuracy: 84.6 -> 84.0 | Hurt |
| Change threshold from δ=0.7 to δ=0.6 (ablation) | RSD with Qwen2.5-Math-Instruct-7B target, 1.5B draft, 1.5B PRM · MATH500 | accuracy: 82.6 -> 80.4 | Hurt |
| Change threshold from δ=0.7 to δ=0.8 (ablation) | RSD with Qwen2.5-Math-Instruct-7B target, 1.5B draft, 1.5B PRM · MATH500 | accuracy: 82.6 -> 82.6 | No clear effect |
| Change threshold from δ=0.7 to δ=0.9 (ablation) | RSD with Qwen2.5-Math-Instruct-7B target, 1.5B draft, 1.5B PRM · MATH500 | accuracy: 82.6 -> 81.2 | Hurt |
| Use reward-based binary step weighting instead of constant weighting (ablation) | RSD (draft 1.5B, target 7B, PRM 7B) · MATH500 | accuracy | Helped |
| Merge target model with PRM instead of keeping them separate (ablation) | RSD (7B target, 1.5B draft, 7B PRM) · Average across six reasoning benchmarks | accuracy: 66.1~(+0.8) -> 66.7~(+1.4) | Helped |
| Merge draft model with PRM instead of keeping them separate (ablation) | RSD (7B target, 1.5B draft, 1.5B PRM) · Average across six reasoning benchmarks | accuracy: 65.9~(+0.6) -> 65.0~(-0.3) | Hurt |
Surprising findings
- Test-time scaling methods (majority voting, Best-of-N) with the draft model consistently underperform the single target model on average.
- Standard speculative decoding often underperforms the target model in practice, which the authors attribute to floating-point errors.
Limitations
- Relies on a general-purpose PRM not specialized for the draft model, which may limit acceptance quality.
- The delta threshold is tuned per task for best results; a fixed delta does not always give the best accuracy.
- Extension to general-domain generation depends on an outcome reward model, with the authors noting no general-domain PRM exists.
Experimental setup
Datasets: GSM8K MATH500 MMLU STEM OlympiadBench GaoKao-2023-En GPQA Diamond Minerva Math CN Middle School 24 College Math AlpacaEval
Benchmarks: MATH500 GSM8K GaoKao-2023-En OlympiadBench GPQA Diamond MMLU STEM Minerva Math CN Middle School 24 College Math AlpacaEval
Metrics: accuracy win rate FLOPs
Baselines: Single target model Majority Voting Best-of-N Speculative decoding (SD) Beam Search Process Best-of-N
Builds on: speculative decoding best-of-N beam search Skywork-o1-Open-PRM
Closest papers by meaning
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.93)
- Guided Speculative Inference for Efficient Test-Time Alignment of LLMs (ICLR 2026 · similarity 0.93)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs (ICLR 2026 · similarity 0.91)
- SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.