Research map / Large language models / Reasoning and chain of thought

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Speculative decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EfficientRollout, a system-aware self-speculative decoding framework for RL rollouts that uses a per-step W4 RTN-quantized copy of the target as drafter, a roofline-based toggle for when SD is enabled, and adaptive draft length, cutting rollout time up to 24.2% and step time up to 15.6%.

Problem

Rollout generation in RL post-training for LLMs is a dominant latency bottleneck because sequential autoregressive decoding is slow; existing speculative decoding methods for RL rollouts either have low block efficiency or require drafter pretraining and online adaptation, and SD can be slower than

Approach

Induce a 4-bit weight-quantized drafter from the current target policy at every training step (self-speculative decoding, no separate drafter training), enable SD only when a roofline model predicts speedup, and adapt the draft length to measured block efficiency.

Key results

• EfficientRollout reduces rollout generation time by 24.2% and step time by 15.6% on Qwen2.5-7B versus veRL (AR), with alpha 98.1%.
• On Qwen2.5-14B, step time drops 10.6% while the quantized self-SD baseline alone slows step time by 0.9%.
• Learned auxiliary EAGLE3 drafter raises Llama3.1-8B step time by 25.6% with alpha of only 54.3%.

Surprising findings

• Quantized self-SD without the toggle slows Qwen2.5-14B step time by 0.9%, showing the drafter alone is not sufficient for speedup.
• Learned auxiliary drafting (EAGLE3) slows Llama3.1-8B step time by 25.6% despite gains on other models.

Limitations

• Gains are model-dependent; RTN quantization may not yield strong early drafters for all model families.
• Only chain verification is used; tree verification is not explored.
• Experiments are on math RLVR with 8k max response length on A100/H100 clusters.

Experimental setup

Datasets: SimpleRL-8k-hard SimpleRL-8k-medium DAPO-Math-17K MATH

Benchmarks: SimpleRL-8k-hard SimpleRL-8k-medium DAPO-Math-17K

Metrics: rollout generation time training step time block efficiency per-token acceptance rate preparation time training reward

Baselines: veRL (AR) Spec-RL EAGLE3-based learned auxiliary drafter quantized self-SD

Builds on: GRPO EAGLE3 Spec-RL Marlin veRL vLLM

Closest papers by meaning

• Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning (ICLR 2026 · similarity 0.90)
• Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing (NeurIPS 2026 · similarity 0.89)
• When RL Meets Adaptive Speculative Training: A Unified Training-Serving System (ICML 2026 · similarity 0.89)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.89)
• AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.89)
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.88)