Research map / Reinforcement learning / RL from feedback
OPPO: Accelerating PPO-based RLHF via Pipeline Overlap
Kaizhuo Yan, YingJie Yu, Yifan Yu, Haizhong Zheng, Fan Lai
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intra-step overlap (stream actor chunks to reward model) alone (ablation) | TRL PPO · Stack-Exchange-Paired, Qwen2.5-7B-Instruct | time to reward 4.17: 4,200 -> 3,500 minutes (1.2x) | Improved |
| add inter-step overlap (overcommit prompts, defer stragglers) alone (ablation) | TRL PPO · Stack-Exchange-Paired, Qwen2.5-7B-Instruct | time to reward 4.17: 4,200 -> 2,700 minutes (1.6x) | Improved |
| add intra-step overlap alone (ablation) | TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instruct | time to reward 5.12: 13,000 -> 10,000 minutes (1.3x) | Improved |
| add inter-step overlap alone (ablation) | TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instruct | time to reward 5.12: 13,000 -> 6,300 minutes (2.06x) | Improved |
| combine both overlaps (full OPPO) | TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instruct | time to reward 5.12: 13,000 -> 5,200 minutes (2.5x) | Improved |
| use dynamic Delta instead of fixed Delta (4 or 8) (ablation) | OPPO · RLHF training convergence | convergence speed and reward stability | Improved |
| use fixed Delta=8 instead of Delta=4 (ablation) | OPPO · RLHF training convergence | convergence speed | Improved |
| use moderate chunk size (500 tokens) vs 100 or 1000-3000 tokens (ablation) | OPPO intra-step streaming · step speed across model sizes | step time | Improved |
| replace TRL PPO with OPPO for training | Qwen2.5-3B · ARC/HellaSwag/TruthfulQA/GSM8K downstream | accuracy: avg 64.19 -> 64.67 | Improved |
| replace TRL PPO with OPPO for training | Qwen2.5-7B · ARC/HellaSwag/TruthfulQA/GSM8K downstream | accuracy: avg 72.93 -> 72.95 | No clear effect |
| use OPPO instead of VeRL w/ DP | PPO RLHF framework · per-step latency | mean latency (s): 125.36 -> 99.84 | Improved |
| use OPPO instead of TRL in multi-node setting | PPO RLHF · Stack-Exchange-Paired multi-node | mean step latency (s): 498.30 -> 111.08 | Improved |
Closest papers by meaning
- DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.92)
- ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models (ICML 2024 · similarity 0.91)
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (ICML 2024 · similarity 0.91)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.89)
- MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
