Research map / Reinforcement learning / RL from feedback

DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DORA, an asynchronous RL rollout system that keeps multiple policy versions concurrently, each trajectory generated under one version, with load-balancing orchestration and zero-re-prefill KV-cache migration, reporting up to 2.12x end-to-end throughput over synchronous training.

Problem

Asynchronous RL post-training for LLMs suffers from a long-tail dilemma: rollout throughput is bottlenecked by the longest trajectories, and existing fixes either incur re-prefill overhead (partial rollout) or discard long trajectories (replication-based), biasing training.

Approach

Maintain multiple policy versions concurrently in the rollout cluster, tag each prompt with one version so every trajectory is generated end-to-end under a single policy, then dynamically repartition DP groups and migrate requests with KV-cache transfer (no re-prefill) because same-version KV states are equivalent.

Key results

• On 64 GPUs DORA reaches 23,327 tokens/s, 1.65x over synchronous training and 1.17x over partial rollout; on 128 GPUs 34,135 tokens/s (2.12x and 1.11x).
• Rollout-stage acceleration of 8.2x on 64 GPUs (14.9 min to 1.8 min) and 5.9x on 128 GPUs.
• In production on a ~500B MoE model with 64K responses, DORA gives 3.6x rollout speedup on math/tool reasoning and up to 6.2x on agentic training (Tau2-bench, Vita).

Surprising findings

• Reported KV-cache reuse ablation yields only about 9% rollout speedup (183 s to 166 s) on dense 32B.
• Stratified analysis suggests stale (staleness-1) responses are longer and harder, so naive staleness comparisons are confounded by selection bias.

Limitations

• Staleness bound K must be manually configured; relies on PPO clipping for off-policy bias.
• Experiments are within an in-house framework; not benchmarked against public systems such as veRL or AReaL.
• MoE scaling validated primarily on production data and cluster, not open-source MoE models.

Experimental setup

Datasets: DAPO-Math-17K

Benchmarks: Tau2-bench Vita

Metrics: end-to-end throughput (tokens/s) average step time (min) rollout speedup mean training reward pass rate

Baselines: Synchronous training One-step off-policy Partial rollout

Builds on: GRPO DAPO PPO vLLM Megatron-LM

Closest papers by meaning

• AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.91)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.91)
• ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.90)
• Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.90)
• AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.89)
• Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.89)