Research map / Reinforcement learning / RL from feedback
Robust Reward Alignment via Hypothesis Space Batch Cutting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HSBC, a robust preference-based reward learning method that iteratively cuts a reward hypothesis space using batches of disagreement-queried preferences and a conservative voting threshold, evaluated with MPPI control on simulated and real-human tasks.
Problem
Preference-based reward alignment is fragile to unknown false (erroneous or adversarial) human preferences, and existing methods lack provable robustness and query-complexity guarantees.
Approach
Treat reward learning as iterative hypothesis-space cutting: each batch of disagreement-selected preference queries yields a batch cut via a voting function, and a conservativeness level gamma relaxes the cut threshold so that up to a fraction of false labels in a batch cannot remove the true reward, without explicitly identifying false labels.
Key results
- Under 30% false preferences on Walker, HSBC achieves 417.2 ± 12.3 vs. 277.0 ± 62.3 for the PEBBLE-style baseline (oracle 472.9).
- Under 30% false preferences on Cartpole, HSBC achieves 111.3 ± 16.8 vs. 42.8 ± 23.3 for the baseline (oracle 148.6).
- With 0% false preferences, HSBC is comparable to the baseline (e.g., Walker 459.4 vs. 444.4; Cartpole 137.8 vs. 143.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HSBC instead of the PEBBLE-style baseline | Reward alignment · Six control tasks, 20% and 30% false preference rates | evaluation reward | Helped |
| Use HSBC instead of the PEBBLE-style baseline | Reward alignment · Six control tasks, 0% false preference rate | evaluation reward | Mixed |
| Set conservativeness level γ to 0 rather than accounting for 20% false preferences (ablation) | HSBC on Walker · Walker, 20% false preference rate | evaluation reward | Hurt |
| Set conservativeness level γ to 30% rather than matching the actual 20% false rate (ablation) | HSBC on Walker · Walker, 20% false preference rate | evaluation reward | No clear effect |
| Use disagreement-based query selection rather than no disagreement threshold (η = 0) (ablation) | HSBC on Walker · Walker, error-free preferences | learning convergence | Helped |
| Vary preference batch size among N = 5, 10, 15, 20 (ablation) | HSBC on Walker · Walker, 20% false preference rate | learning performance | No clear effect |
| Increase ensemble size among M = 4, 8, 16, 32 (ablation) | HSBC on Walker · Walker, 20% false preference rate | learning performance | Helped |
| Use HSBC with a Stoc teacher rather than ideal oracle preferences | HSBC on Cartpole · Cartpole, Stoc teacher | evaluation reward: 148.6 -> 93.6 ±30.7 | Hurt |
Surprising findings
- The Stoc (stochastic) B-Pref teacher degrades HSBC noticeably (93.6 vs. 148.6 oracle on Cartpole), which the authors attribute to near-equal trajectory rewards late in learning.
- Setting the conservativeness level gamma to 0 (no conservative cut) degrades performance substantially under 20% false preferences.
Limitations
- Stoc-type noisy teachers hurt performance significantly.
- Higher conservativeness level gamma increases query complexity since less of the hypothesis space is cut per iteration.
- Experiments use simulated humans mostly; real-human study is small (four volunteers, Cartpole and Walker only) and uses gamma=40%.
Experimental setup
Datasets: simulated preferences from ground-truth rewards in MJX dm-control and dexterous/quadruped tasks real human preferences from four volunteers
Benchmarks: Cartpole-Swingup Walker-Walk Humanoid-Standup Dexman-Cube Dexman-Bunny Go2-Standup
Metrics: ground-truth episode reward Pearson correlation between learned and ground-truth reward
Baselines: PEBBLE RIME SURF MAE t-CE
Builds on: PEBBLE Bradley-Terry preference model MPPI disagreement coefficient active learning
Closest papers by meaning
- Rectifying Shortcut Behaviors in Preference-based Reward Learning (NeurIPS 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking (ICLR 2025 · similarity 0.89)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.89)
- PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation (ICML 2024 · similarity 0.89)
- Adaptive Preference Scaling for Reinforcement Learning with Human Feedback (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.