AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

The Art of Scaling Reinforcement Learning Compute for LLMs

Fnu Devvrit, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, Rishabh Agarwal

ICLR 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 99 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use PipelineRL-k instead of PPO-off-policy-k (ablation)asynchronous RL baseline · RL validation (math)compute efficiency BImproved
use GSPO or CISPO instead of DAPO loss (ablation)RL loss · RL validation (math)asymptotic pass rate AImproved
FP32 precision at LM head (ablation)RL baseline · RL validation (math)asymptotic pass rate A: 0.52 -> 0.61Improved
prompt-average loss aggregation instead of sample or token average (ablation)RL baseline · RL validation (math)asymptotic pass rate AImproved
batch-level instead of prompt-level or no advantage normalization (ablation)RL baseline · RL validation (math)pass rateNo clear effect
zero-variance filtering (effective batch) (ablation)RL baseline · RL validation (math)asymptotic pass rate AImproved
no-positive-resampling of prompts with pass rate >= 0.9 (ablation)RL baseline · RL validation (math)asymptotic pass rate AImproved
length penalty instead of interruption for length control (ablation)ScaleRL · RL validation (math), 16k GPU-hourspass rateNo clear effect
CISPO instead of DAPO loss in ScaleRL (ablation)ScaleRL · leave-one-outcompute efficiency B: 1.77 -> 2.01Improved
leave-one-out of individual ScaleRL components (ablation)ScaleRL · leave-one-out at 16k GPU-hoursasymptotic pass rate A: within +/-0.02No clear effect
increase generation length from 14k to 32k (ablation)ScaleRL · RL validation (math)asymptotic pass rate A: 0.610 -> 0.645Improved
increase batch size from 512 to 2048 prompts (ablation)ScaleRL · RL validation (math)asymptotic pass rate A: 0.605 -> 0.645Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy