Research map / Reinforcement learning / RL from feedback

AsyncOPD: How Stale Can On-Policy Distillation Be?

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Systematic study of staleness in asynchronous on-policy distillation with finite teacher-score caches, finding reverse-KL needs recomputed advantages without clipping and multi-sample MC, and presents the open-sourced AsyncOPD pipeline giving 1.6x-3.8x throughput over synchronous training.

Problem

Asynchronous on-policy distillation (OPD) reduces rollout bottlenecks but introduces stale-policy data, and finite teacher-score caches make it unclear how existing asynchronous RL stabilizers and OPD estimators behave under staleness.

Approach

Shows KL direction determines staleness sensitivity (forward KL robust, reverse KL vulnerable); for reverse KL, recomputing the advantage at learner time without clipping beats PPO-style and async-RL surrogates; multi-sample Monte Carlo with cached teacher scores reduces one-sample variance while keeping importance-sampling correctability.

Key results

• AsyncOPD improves training throughput by 1.6x to 3.8x over strict synchronous training while reaching comparable accuracy (abstract).
• Multi-sample MC with 64 samples leaves 1.49% of one-sample local variance at staleness 128 on Qwen3-4B-Base.
• Recomputing the reverse-KL advantage at learner time without clipping gives the best staleness-sensitivity slope (AIME24 -0.69 vs -3.99 for old advantage without clipping).

Surprising findings

• Clipping hurts the recomputed-advantage surrogate, while advanced asynchronous RL surrogates (decoupled PPO, M2PO) do not outperform the simple unclipped OPD/IS surrogate.
• Reverse KL starts higher than forward KL at zero staleness but degrades faster and is eventually overtaken.

Limitations

• Only sparse and Monte Carlo estimators studied; dense full-vocabulary KL not evaluated in the asynchronous setting.
• Experiments limited to a single 8-GPU node.
• Evaluation focused on math benchmarks (AIME24, AIME25, AMC) with one teacher model.

Experimental setup

Datasets: DeepMath

Benchmarks: AIME24 AIME25 AMC

Metrics: Avg@32 accuracy training throughput (tok/s) pipeline overlap staleness-sensitivity slope MC variance ratio

Baselines: Strict synchronous training Two-step-off scheduling Decoupled PPO M2PO PPO-style clipped surrogate Sparse top-k forward KL

Builds on: GKD MiniLLM PPO Decoupled PPO M2PO AReaL VeRL

Closest papers by meaning

• Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.91)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.91)
• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.90)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
• Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.90)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.90)