Research map / Reinforcement learning / RL from feedback

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

ICLR 2025 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes asynchronous off-policy RLHF that separates vLLM-based generation from training on separate GPUs, studying off-policyness robustness of RLHF losses (Online DPO most robust) and speedups at TLDR, chatbot, and GSM8k scales.

Problem

Synchronous on-policy RLHF is computationally inefficient because generation with training libraries is slow and GPUs idle; it is unclear how much off-policy data RLHF methods can tolerate for asynchronous training.

Approach

Separate generation (vLLM on one GPU) from training (other GPUs) so they run asynchronously, training on samples from the previous policy iteration; find Online DPO is the most robust RLHF loss to off-policyness and that robustness grows with policy scale.

Key results

• Async Online DPO matches sync win-rate on TLDR at 2.8B while running ~25% faster; with K=4 sampling, training-bound setup reaches the same win-rate about 2.5x faster at cost of higher KL.
• On No Robots with LLaMA 3.1 8B, async Online DPO reaches 57.2% GPT-4o win rate, identical to sync, while running 38% faster (142 vs 230 min).
• On GSM8k with Rho-1B, async Online DPO reaches 52.6% pass@1 vs 52.2% for sync, running 68% faster (129 vs 218 minutes).

Measured effects reported in the paper

Surprising findings

• Online DPO is the only tested RLHF loss that still learns meaningfully at N=64 off-policy mini-batches, while PPO and RLOO degrade sharply with off-policyness.
• Scaling the reward model does not improve off-policy robustness, whereas scaling the policy model does.

Limitations

• Async speedups are reduced by GIL contention and parameter-communication overhead; practical runtime falls short of ideal.
• Off-policy PPO/RLOO degrade quickly; only Online DPO is robust, and tradeoffs with KL appear under compute optimizations.
• Experiments use relatively small Pythia models on TLDR and a synthetic GPT-4o-labelled preference setup.

Experimental setup

Datasets: TLDR summarization No Robots GSM8k

Benchmarks: TLDR summarization No Robots test set GSM8k

Metrics: gold win rate KL (perplexity) pass@1 compute time

Baselines: PPO RLOO Best-of-2 SFT Sync Online DPO Sync PPO Sync RLOO CoPG

Builds on: Online DPO Cleanba PPO REINFORCE Leave-One-Out (RLOO) vLLM

Closest papers by meaning

• AsyncOPD: How Stale Can On-Policy Distillation Be? (NeurIPS 2026 · similarity 0.91)
• AREAL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning (NeurIPS 2025 · similarity 0.91)
• Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training (NeurIPS 2025 · similarity 0.90)
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training (NeurIPS 2026 · similarity 0.90)
• Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction (NeurIPS 2026 · similarity 0.89)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)