Research map / Reinforcement learning / RL from feedback
Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes policy-gradient RLHF algorithms (PO-RLHF) with active trajectory-comparison queries for linear and neural function approximation, extending PC-PG with a trajectory-level elliptical potential analysis to bound reward-estimation error.
Problem
Theoretical justification for why RLHF needs only a small amount of human feedback, and query/sample complexity of policy-based RLHF with exploration when rewards are unobserved and only trajectory comparisons are given.
Approach
Replaces PC-PG's known reward with a reward estimated from Bradley-Terry trajectory comparisons against a fixed baseline policy, and bounds reward-estimation error via a trajectory-level elliptical potential lemma so that the number of queries is a small fraction of total samples (roughly 1/T).
Key results
- PO-RLHF (algorithm for linear setting) learns near-optimal policy on Bidirectional Lock without observing true rewards, achieving performance comparable to PC-PG with a few more samples and a small number of queries.
- Query-to-total-sample ratio scales as 1/T, so the number of human queries becomes a small fraction of total sample complexity as policy optimization iterations T grow.
- Theorem bounds suboptimality by terms in epsilon_bias, 1/sqrt(T), (M_SGD)^(-1/4), M_HF^(-1/4), and 1/N, with the transfer error remaining even with infinite samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace direct reward observations with trajectory preference feedback | PC-PG on Bidirectional Lock · Bidirectional Lock | normalized suboptimality | No clear effect |
| increase policy-optimization iterations T | PO-RLHF (linear) · Bidirectional Lock | query-complexity-to-sample-complexity ratio | Helped |
Limitations
- Experiments limited to a small tabular-style Bidirectional Lock environment with one-hot features.
- Requires a baseline policy satisfying a covariance assumption (Assumption on baseline policy) and regularity assumptions on state-action distributions.
- Transfer function approximation error does not vanish even with infinite samples.
Experimental setup
Datasets: Bidirectional Lock environment (simulated)
Benchmarks: Bidirectional Lock
Metrics: normalized suboptimality sample complexity query complexity
Baselines: PC-PG
Builds on: PC-PG natural policy gradient Bradley-Terry model elliptical potential lemma
Closest papers by meaning
- Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference (ICLR 2025 · similarity 0.92)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.92)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.92)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.91)
- Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.91)
- A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.