AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?

Haizhong Zheng, Jiawei Zhao, Beidi Chen

ICLR 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
constrain second moment of importance log-ratio (M2PO) instead of epsilon-clippingGRPO · 8 math benchmarks, staleness 256, Qwen3-Base-4Baverage accuracy: 40.1 -> 51.3Improved
M2PO at staleness 256 vs on-policy GRPOM2PO · Qwen3-Base-1.7Baverage accuracy: 33.0 -> 36.6Improved
increase staleness from 0 to 256 in GRPO (ablation)GRPO · Qwen3-Base-4Baverage accuracy: 50.7 -> 40.1Hurt
remove trust region (no clipping) under staleness 256 (ablation)GRPO · Qwen2.5-Math-7BaccuracyMixed
use sequence-level clipping (GSPO) instead of token clipping under staleness 256 (ablation)GRPO · Qwen3-Base-4Baverage accuracy: 40.1 -> 43.2Mixed
use GSPO under staleness 256 on Qwen2.5-Math-7B and Qwen3-Base-1.7B (ablation)GRPO · Qwen2.5-Math-7Baverage accuracy: 45.7 -> 44.7No clear effect
use AREAL, TOPR, GPPO, CISPO under staleness 256 (ablation)GRPO variants · Qwen modelstraining stabilityHurt
replace epsilon-clipping with M2PO masking (ablation)GRPO · clipping ratio on Qwen2.5-32B, s=256clipping ratio (%): 1.22 -> 0.06Improved
vary M2PO threshold tau (ablation)M2PO · Llama-3.2-3B and Qwen2.5-Math-7BaccuracyNo clear effect
vary staleness 0 to 256 for M2PO (ablation)M2PO · Qwen2.5-Math-7BaccuracyNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy