Research map / Reinforcement learning / RL from feedback
Hindsight PRIORs for Reward Learning from Human Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hindsight PRIOR, which uses attention weights from a Transformer-XL world model to estimate state importance and redistributes predicted returns as an auxiliary reward target for preference-based RL.
Problem
Preference-based RL reward learning suffers from the credit assignment problem: many reward functions fit the trajectory-level preference labels but misassign per-state rewards, requiring large amounts of human feedback and yielding misaligned rewards.
Approach
Use attention weights of a transformer-based forward dynamics model (TWM) as a proxy for state-action importance, and redistribute the predicted trajectory return according to these weights as an auxiliary MSE target for the reward model, added to the standard cross-entropy preference loss.
Key results
- Hindsight PRIOR achieves mean normalized success 0.66 on MetaWorld vs 0.56 for PEBBLE and 0.55 for MRN, and normalized return 0.59 on DMC vs 0.55 for SURF (best baseline).
- Improvements are reported as statistically significant (p<0.05) against PEBBLE, SURF, RUNE, and MRN on both suites.
- With 20% labeller mistakes, Hindsight PRIOR outperforms PEBBLE with no mistakes on window-open.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add state-importance-guided predicted-return redistribution to PEBBLE | PEBBLE · MetaWorld | normalized success rate: 0.56 $\pm$ 0.007 -> 0.66$\pm$ 0.002 | Helped |
| Add state-importance-guided predicted-return redistribution to PEBBLE | PEBBLE · DMC | normalized return: 0.48 $\pm$ 0.018 -> 0.59$\pm$ 0.003 | Helped |
| Use uniform importance for return redistribution instead of forward-dynamics attention (ablation) | Hindsight PRIOR · MetaWorld and DMC | policy performance | Hurt |
| Apply min-max normalization and softmax to attention weights (ablation) | Hindsight PRIOR · MetaWorld and DMC | policy performance | No clear effect |
| Replace the PRIOR loss with a bisimulation auxiliary objective (ablation) | Hindsight PRIOR · MetaWorld and DMC | policy performance | Hurt |
| Increase preference trajectory length from 50 to 200 (ablation) | Hindsight PRIOR · MetaWorld and DMC | policy performance: 50 versus 200 | No clear effect |
| Combine Hindsight PRIOR with SURF (ablation) | Hindsight PRIOR · MetaWorld | policy performance | Mixed |
| Remove preference feedback by making the PRIOR loss dominate (ablation) | Hindsight PRIOR · MetaWorld | policy learning | Hurt |
| Add state-importance-guided reward learning | PEBBLE · Window Open with 20% incorrect feedback | policy performance: 20% | Helped |
Surprising findings
- Setting lambda very large so reward learns only from the prior loss yields no usable policy, showing redistribution alone is insufficient.
- Uniform return redistribution (RVAR) performs roughly on par with PEBBLE, and bisimulation-based dynamics awareness adds only marginal gains.
Limitations
- Assumes states important to the world model are also important to a human evaluator; human attention may differ.
- Evaluated only with synthetic oracle feedback from hand-coded rewards rather than real humans.
- Gains on some tasks are absent (e.g., quadruped-walk) and combining with SURF gives no extra gain over PRIOR alone.
Experimental setup
Datasets: MetaWorld DeepMind Control Suite Atari Montezuma's Revenge
Benchmarks: MetaWorld DeepMind Control Suite
Metrics: normalized success rate normalized return paired t-test p-value
Baselines: PEBBLE SURF RUNE MRN SAC RVAR BISIM
Builds on: PEBBLE Transformer-based World Models (TWM) return redistribution (RRD) Bradley-Terry model SURF
Closest papers by meaning
- Towards Theoretical Understanding of Sequential Decision Making with Preference Feedback (ICML 2025 · similarity 0.89)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.88)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.88)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.88)
- Going Beyond Heuristics by Imposing Policy Improvement as a Constraint (NeurIPS 2024 · similarity 0.88)
- Policy-labeled Preference Learning: Is Preference Enough for RLHF? (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.