Research map / Large language models / Reasoning and chain of thought
AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents AReaL, a fully asynchronous RL system for LLM reasoning that decouples streaming rollout generation from training, paired with a staleness-controlled decoupled PPO objective.
Problem
Synchronous RL systems for large reasoning models waste GPU time because generation must wait for the longest output in each batch, and asynchronous training introduces data staleness that can degrade PPO.
Approach
Decouple rollout workers from trainer workers with interruptible, continuously running generation; control data staleness via a maximum-staleness constraint; and use a decoupled PPO objective that separates behavior and proximal policies so samples from multiple policy versions remain usable.
Key results
- AReaL reduces end-to-end training hours up to 2.77x versus synchronous systems with matched or slightly improved AIME24 scores (e.g., 7B: 63.1 vs 63.0 with 25.4 vs 57.7 hours).
- On 14B code model, AReaL reaches 58.1 LiveCodeBench vs 56.7 for synchronous AReaL and 57.9 for verl, in 21.9 hours vs 48.8 and 44.4.
- Naive PPO with staleness degrades performance (e.g., AIME24 23.3 at eta=4), while the decoupled objective restores it (42.2 at eta=4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace synchronous RL with fully asynchronous AReaL | R1-Distilled-Qwen 1.5B · AIME24 | training hours: 41.0 -> 14.8 | Helped |
| replace synchronous RL with fully asynchronous AReaL | R1-Distilled-Qwen 1.5B · AIME24 | pass@1 accuracy: 42.0 -> 42.2 | Mixed |
| replace synchronous RL with fully asynchronous AReaL | R1-Distilled-Qwen 7B · AIME24 | training hours: 57.7 -> 25.4 | Helped |
| replace synchronous RL with fully asynchronous AReaL | R1-Distilled-Qwen 14B · LiveCodeBench (8/1/24-2/1/25) | pass@1 accuracy: 56.7 -> 58.1 | Helped |
| replace synchronous RL with fully asynchronous AReaL | R1-Distilled-Qwen 32B · LiveCodeBench (8/1/24-2/1/25) | pass@1 accuracy: 61.2 -> 61.0 | Hurt |
| add the decoupled PPO objective (ablation) | PPO with max staleness 4 · AIME24 | accuracy: 23.3 -> 42.2 | Helped |
| add the decoupled PPO objective (ablation) | PPO with max staleness 8 · AIME24 | accuracy: 35.7 -> 41.0 | Helped |
| allow max staleness 4 instead of zero staleness (ablation) | PPO without decoupled objective · AIME24 | accuracy: 42.0 -> 23.3 | Hurt |
| use unbounded instead of zero staleness (ablation) | Decoupled PPO · AIME24 | accuracy: 42.0 -> 36.9 | Hurt |
| replace standard micro-batching with dynamic batching (ablation) | PPO training across model sizes · Various model sizes | training throughput: 30% | Helped |
| enable interruptible generation (ablation) | AReaL rollout workers · 1.5B and 7B models on 4 nodes | generation throughput: 12% and 17% | Helped |
Surprising findings
- Naive PPO with even slight staleness significantly degrades final performance, while the decoupled objective recovers near-oracle results.
- Moderate staleness (eta up to 8) has minimal impact on final performance while accelerating training.
Limitations
- Evaluated only on single-step math and coding tasks; multi-turn and agentic settings left for future work.
- Inference-to-training device ratio fixed heuristically at 3:1 rather than optimized or dynamically adjusted.
- Single trial with a fixed seed; no error bars for large-scale experiments.
Experimental setup
Datasets: DeepScaleR math dataset DeepCoder code dataset AIME24 AIME25 AMC23 MATH500 LiveCodeBench
Benchmarks: AIME24 AIME25 AMC23 MATH500 LiveCodeBench Codeforces CodeContests
Metrics: pass@1 accuracy training hours effective training throughput (tokens/s)
Baselines: verl (synchronous) Sync. AReaL DeepScaleR-1.5B DeepCoder-14B naive PPO with staleness
Builds on: PPO GRPO DeepScaleR SGLang Megatron-Core
Closest papers by meaning
- DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.91)
- Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.91)
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.91)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.91)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.91)
- Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.