Research map / Reinforcement learning / RL from feedback
RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RA-PbRL, a provably efficient preference-based RL algorithm optimizing nested or static quantile (CVaR) risk objectives under one-episode trajectory-level preference feedback, with sublinear regret bounds and small tabular and MuJoCo experiments.
Problem
Standard PbRL/RLHF optimizes risk-neutral mean reward, and standard risk-aware RL objectives cannot be computed when only one trajectory-level preference (one-episode reward) is observed.
Approach
Shows nested and static quantile risk objectives are computable under trajectory-level preference feedback, expands the state space to make the process history-dependent, and extends the PbOP confidence-set/optimism algorithm to optimize both objectives with regret guarantees.
Key results
- Theorems give sublinear-in-K regret upper bounds for both nested and static quantile risk objectives under one-episode-reward PbRL.
- Empirically, RA-PbRL regret initially grows linearly in K and becomes sublinear after a threshold; its bad-scenario regret is significantly lower than PbOP and ICVaR-RLHF in the tabular setting (50 trials).
- Counterexample shows two policies with identical reward distributions can have different nested CVaR (0.33 vs 0.36 at alpha=0.2) while static CVaR is equal (0.41).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase α | RA-PbRL · Tabular MDP; static and nested CVaR | regret | Helped |
| Use smaller α (greater risk aversion) | RA-PbRL · Tabular MDP; static and nested CVaR | regret variance | Hurt |
| Use RA-PbRL rather than the compared algorithms | Risk-aware PbRL algorithms · Tabular MDP; bad scenarios | regret | Helped |
| Optimize nested rather than static CVaR | πA versus πB · Constructed MDP, α=0.2 | CVaR: 0.33 vs 0.36 | Mixed |
Surprising findings
- Two policies with identical trajectory reward distributions can have different nested CVaR values, showing nested and static measures diverge.
Limitations
- Feedback limited to pairwise comparisons of two trajectories; n-wise comparisons not considered.
- Reward functions assumed linear in trajectory embedding; risk measure generality limited by assumptions.
- Gap between upper and lower regret bounds remains; experiments are small-scale tabular and a linear-approximation MuJoCo setup.
Experimental setup
Datasets: simulated tabular MDP MuJoCo HalfCheetah
Benchmarks: tabular MDP (H=6, S=4, A=3) MuJoCo HalfCheetah
Metrics: cumulative static CVaR regret cumulative nested CVaR regret
Baselines: PbOP ICVaR-RLHF TRC
Builds on: PbOP ICVaR-RL Bradley-Terry model
Closest papers by meaning
- Provable Reward-Agnostic Preference-Based Reinforcement Learning (ICLR 2024 · similarity 0.92)
- Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF (ICLR 2025 · similarity 0.90)
- Projection Optimization: A General Framework for Multi-Objective and Multi-Group RLHF (ICML 2025 · similarity 0.89)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.89)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.89)
- Decoding Global Preferences: Temporal and Cooperative Dependency Modeling in Multi-Agent Preference-Based Reinforcement Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.