AI Paper Atlas

Research map / Reinforcement learning / RL from feedback

OPPO: Accelerating PPO-based RLHF via Pipeline Overlap

Kaizhuo Yan, YingJie Yu, Yifan Yu, Haizhong Zheng, Fan Lai

ICLR 2026 · Reinforcement learning › RL from feedback · approach: Theory of RLHF

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add intra-step overlap (stream actor chunks to reward model) alone (ablation)TRL PPO · Stack-Exchange-Paired, Qwen2.5-7B-Instructtime to reward 4.17: 4,200 -> 3,500 minutes (1.2x)Improved
add inter-step overlap (overcommit prompts, defer stragglers) alone (ablation)TRL PPO · Stack-Exchange-Paired, Qwen2.5-7B-Instructtime to reward 4.17: 4,200 -> 2,700 minutes (1.6x)Improved
add intra-step overlap alone (ablation)TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instructtime to reward 5.12: 13,000 -> 10,000 minutes (1.3x)Improved
add inter-step overlap alone (ablation)TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instructtime to reward 5.12: 13,000 -> 6,300 minutes (2.06x)Improved
combine both overlaps (full OPPO)TRL PPO · Stack-Exchange-Paired, Qwen2.5-3B-Instructtime to reward 5.12: 13,000 -> 5,200 minutes (2.5x)Improved
use dynamic Delta instead of fixed Delta (4 or 8) (ablation)OPPO · RLHF training convergenceconvergence speed and reward stabilityImproved
use fixed Delta=8 instead of Delta=4 (ablation)OPPO · RLHF training convergenceconvergence speedImproved
use moderate chunk size (500 tokens) vs 100 or 1000-3000 tokens (ablation)OPPO intra-step streaming · step speed across model sizesstep timeImproved
replace TRL PPO with OPPO for trainingQwen2.5-3B · ARC/HellaSwag/TruthfulQA/GSM8K downstreamaccuracy: avg 64.19 -> 64.67Improved
replace TRL PPO with OPPO for trainingQwen2.5-7B · ARC/HellaSwag/TruthfulQA/GSM8K downstreamaccuracy: avg 72.93 -> 72.95No clear effect
use OPPO instead of VeRL w/ DPPPO RLHF framework · per-step latencymean latency (s): 125.36 -> 99.84Improved
use OPPO instead of TRL in multi-node settingPPO RLHF · Stack-Exchange-Paired multi-nodemean step latency (s): 498.30 -> 111.08Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy