Research map / Reinforcement learning / RL from feedback
Efficient RL Training for LLMs with Experience Replay
Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, Rémi Munos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an experience replay buffer to on-policy GRPO | GRPO · OpenR1-Math-220k | compute to reach target accuracy: save up to 40% | Improved |
| add a replay buffer (accuracy at every compute budget) | GRPO · OpenR1-Math-220k | accuracy vs compute | Improved |
| add a replay buffer for training stability | GRPO · OpenR1-Math-220k | training stability | Improved |
| increase buffer size (ablation) | GRPO with replay · Qwen3-0.6B, (W,T)=(6,2) | training speed and peak accuracy | Mixed |
| lower W/T ratio (raise replay ratio 2.2 to 17.6) (ablation) | GRPO with replay · Qwen3-0.6B | maximal accuracy: 2.2 -> 5.6 -> 17.6 | Mixed |
| add replay buffer for pass@k | GRPO · Qwen3-0.6B | pass@k for k>1 | Improved |
| add moderate off-policiness without a buffer (ablation) | GRPO · Qwen3-0.6B | training stability | Improved |
| add positive-bias sampling (delta 0.2 or 0.5) (ablation) | GRPO with replay buffer N=4608 · Qwen3-0.6B, OpenR1-Math-220k | test accuracy | Improved |
| replace GRPO with AsymRE loss (ablation) | replay buffer N=4608 · Qwen3-0.6B, OpenR1-Math-220k | test accuracy | Improved |
| sample buffer uniformly without replacement (ablation) | replay buffer · Qwen3-0.6B | test accuracy | No clear effect |
| add replay buffer for wall-time speed | GRPO async · Qwen3-0.6B / Qwen2.5-7B | wall-time gain | Improved |
| best buffer configs vs best non-buffer configs under learning-rate sweep (ablation) | GRPO · OpenR1-Math-220k | best accuracy per compute | Improved |
Closest papers by meaning
- Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.89)
- Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay (NeurIPS 2025 · similarity 0.88)
- Optimizing Test-Time Compute via Meta Reinforcement Finetuning (ICML 2025 · similarity 0.88)
- SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning (NeurIPS 2026 · similarity 0.87)
- When Does Non-Uniform Replay Matter in Reinforcement Learning? (NeurIPS 2026 · similarity 0.87)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
