Research map / Reinforcement learning / RL from feedback
Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
Haizhong Zheng, Jiawei Zhao, Beidi Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| constrain second moment of importance log-ratio (M2PO) instead of epsilon-clipping | GRPO · 8 math benchmarks, staleness 256, Qwen3-Base-4B | average accuracy: 40.1 -> 51.3 | Improved |
| M2PO at staleness 256 vs on-policy GRPO | M2PO · Qwen3-Base-1.7B | average accuracy: 33.0 -> 36.6 | Improved |
| increase staleness from 0 to 256 in GRPO (ablation) | GRPO · Qwen3-Base-4B | average accuracy: 50.7 -> 40.1 | Hurt |
| remove trust region (no clipping) under staleness 256 (ablation) | GRPO · Qwen2.5-Math-7B | accuracy | Mixed |
| use sequence-level clipping (GSPO) instead of token clipping under staleness 256 (ablation) | GRPO · Qwen3-Base-4B | average accuracy: 40.1 -> 43.2 | Mixed |
| use GSPO under staleness 256 on Qwen2.5-Math-7B and Qwen3-Base-1.7B (ablation) | GRPO · Qwen2.5-Math-7B | average accuracy: 45.7 -> 44.7 | No clear effect |
| use AREAL, TOPR, GPPO, CISPO under staleness 256 (ablation) | GRPO variants · Qwen models | training stability | Hurt |
| replace epsilon-clipping with M2PO masking (ablation) | GRPO · clipping ratio on Qwen2.5-32B, s=256 | clipping ratio (%): 1.22 -> 0.06 | Improved |
| vary M2PO threshold tau (ablation) | M2PO · Llama-3.2-3B and Qwen2.5-Math-7B | accuracy | No clear effect |
| vary staleness 0 to 256 for M2PO (ablation) | M2PO · Qwen2.5-Math-7B | accuracy | No clear effect |
Closest papers by meaning
- ROLLVERIFY: BRIDGING EFFICIENCY AND ACCURACY IN LONG-TAIL ROLLOUT REINFORCEMENT LEARNING (NeurIPS 2026 · similarity 0.91)
- AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.91)
- Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.90)
- VESPO: Variational Sequence-level Soft Policy Optimization for Off-Policy LLM Training (NeurIPS 2026 · similarity 0.90)
- STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens (NeurIPS 2026 · similarity 0.89)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
