Research map / Reinforcement learning / RL from feedback
The Art of Scaling Reinforcement Learning Compute for LLMs
Fnu Devvrit, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, Rishabh Agarwal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PipelineRL-k instead of PPO-off-policy-k (ablation) | asynchronous RL baseline · RL validation (math) | compute efficiency B | Improved |
| use GSPO or CISPO instead of DAPO loss (ablation) | RL loss · RL validation (math) | asymptotic pass rate A | Improved |
| FP32 precision at LM head (ablation) | RL baseline · RL validation (math) | asymptotic pass rate A: 0.52 -> 0.61 | Improved |
| prompt-average loss aggregation instead of sample or token average (ablation) | RL baseline · RL validation (math) | asymptotic pass rate A | Improved |
| batch-level instead of prompt-level or no advantage normalization (ablation) | RL baseline · RL validation (math) | pass rate | No clear effect |
| zero-variance filtering (effective batch) (ablation) | RL baseline · RL validation (math) | asymptotic pass rate A | Improved |
| no-positive-resampling of prompts with pass rate >= 0.9 (ablation) | RL baseline · RL validation (math) | asymptotic pass rate A | Improved |
| length penalty instead of interruption for length control (ablation) | ScaleRL · RL validation (math), 16k GPU-hours | pass rate | No clear effect |
| CISPO instead of DAPO loss in ScaleRL (ablation) | ScaleRL · leave-one-out | compute efficiency B: 1.77 -> 2.01 | Improved |
| leave-one-out of individual ScaleRL components (ablation) | ScaleRL · leave-one-out at 16k GPU-hours | asymptotic pass rate A: within +/-0.02 | No clear effect |
| increase generation length from 14k to 32k (ablation) | ScaleRL · RL validation (math) | asymptotic pass rate A: 0.610 -> 0.645 | Improved |
| increase batch size from 512 to 2048 prompts (ablation) | ScaleRL · RL validation (math) | asymptotic pass rate A: 0.605 -> 0.645 | Improved |
Closest papers by meaning
- Compute-Optimal Scaling for Value-Based Deep RL (NeurIPS 2025 · similarity 0.93)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.92)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.91)
- AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.91)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.91)
- UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
