Research map / Reinforcement learning / RL from feedback
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZPG and ZBCPG, zeroth-order policy gradient methods that estimate value differences from human trajectory preferences, enabling RLHF without reward inference for general stochastic MDPs.
Problem
RLHF typically needs reward inference, and DPO's closed-form optimal-policy relation only holds for bandits or deterministic MDPs; no provable policy-based RLHF without reward inference existed for general stochastic RL problems.
Approach
Estimate local value function differences between a policy and a randomly perturbed policy from trajectory-pair human preferences (inverting the link function), then plug these into a two-point zeroth-order gradient estimator; a block-coordinate variant uses Rademacher perturbations on sampled coordinates.
Key results
- Under Bradley-Terry feedback, last-iterate return is 1.94 for ZPG and 1.91 for ZBCPG versus 1.80 for RM+PPO, 1.71 for Online DPO and -4.13 for DPO.
- Under Weibull feedback with M=1000, ZPG reaches 2.13 and ZBCPG 1.90 versus 1.72 for RM+PPO and -4.25 for DPO.
- Theory: convergence rate O(Hd/T + d^2 sqrt(log M)/sqrt(M) + Hd sqrt(d)/sqrt(N)) to a stationary point, with TMN = O(d^8 H^3 / eps^5 log(d/eps)) human queries.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZPG instead of RM+PPO | GridWorld RLHF · GridWorld, Bradley-Terry feedback | return: 1.80\pm 0.09 -> 1.94 \pm 0.09 | Helped |
| Use ZBCPG instead of RM+PPO | GridWorld RLHF · GridWorld, Bradley-Terry feedback | return: 1.80\pm 0.09 -> 1.91 \pm 0.09 | Helped |
| Use ZPG instead of RM+PPO | GridWorld RLHF · GridWorld, Weibull feedback, M=1000 | return: 1.72\pm 0.09 -> 2.13 \pm 0.09 | Helped |
| Use ZBCPG instead of RM+PPO | GridWorld RLHF · GridWorld, Weibull feedback, M=1000 | return: 1.72\pm 0.09 -> 1.90 \pm 0.09 | Helped |
| Use ZPG instead of RM+PPO | GridWorld RLHF · GridWorld, Weibull feedback, M=100 | return: 1.63\pm 0.09 -> 2.02 \pm 0.08 | Helped |
| Use ZBCPG instead of RM+PPO | GridWorld RLHF · GridWorld, Weibull feedback, M=100 | return: 1.63\pm 0.09 -> 2.16 \pm 0.09 | Helped |
| Reduce human panel size from M=1000 to M=100 (ablation) | ZPG · GridWorld, Weibull feedback | return: 2.13 \pm 0.09 -> 2.02 \pm 0.08 | No clear effect |
| Reduce human panel size from M=1000 to M=100 (ablation) | ZBCPG · GridWorld, Weibull feedback | return: 1.90 \pm 0.09 -> 2.16 \pm 0.09 | No clear effect |
| Increase the number of parallel blocks (ablation) | Distributed ZBCPG · GridWorld, Bradley-Terry feedback | convergence speed | Helped |
| Increase the number of parallel blocks, reducing queries per trajectory pair in each group (ablation) | Distributed ZBCPG · GridWorld, Bradley-Terry feedback | final policy return | Hurt |
Surprising findings
- Vanilla DPO performs far worse than the reward-model-based RM+PPO baseline in the stochastic GridWorld (return around -4 versus about 1.8).
- ZPG with a smaller panel (M=100) under Weibull feedback achieves a return of 2.02, close to the M=1000 result.
Limitations
- Requires many online human queries for accurate gradient estimation, limiting practicality.
- Only local convergence to stationary points is established; no global optimality guarantee.
- Lacks strategic exploration and KL regularization, leading to policy fluctuation; experiments only on a small tabular GridWorld rather than LLMs.
Experimental setup
Datasets: simulated GridWorld trajectories with simulated human panel
Benchmarks: stochastic 5x5 GridWorld
Metrics: last-iterate average return trajectory return over iterations
Baselines: RM+PPO DPO Online DPO
Builds on: DPO zeroth-order SGD (Nesterov and Spokoiny style two-point estimator) PPO online DPO
Closest papers by meaning
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.94)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.93)
- Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.93)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.93)
- Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.93)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.