Research map / Reinforcement learning / RL from feedback
Deep Reinforcement Learning from Hierarchical Preference Design
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HERON, which builds preference labels by comparing RL trajectories with a decision tree over a human-ranked hierarchy of feedback signals, then trains a reward model (or DPO) for policy learning.
Problem
Reward design in RL is hard when feedback signals have different scales and intricate dependencies, or when rewards are sparse and surrogate signals are needed; linear reward engineering requires extensive tuning.
Approach
Instead of weighting feedback signals linearly, rank them by importance and use a hierarchical decision tree (with per-level margins set to the signal's standard deviation) to elicit trajectory preferences, then train a Bradley-Terry reward model on these preferences; the method is scale-invariant and needs no numeric weights.
Key results
- On APPS, HERON reaches Pass@50 of 10.19 vs. 9.81 for CodeRL and 7.62 for PPOCoder.
- On HelpSteer alignment, HERON-DPO achieves an average win rate of 64.57% against the SFT model vs. 57.06% for the best reward-engineering baseline.
- In traffic light control, HERON outperforms linear reward engineering with p<0.005, and in robotics it beats reward engineering on all four tasks (e.g., Ant 1.0 vs 0.88 ground-truth reward).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical preference-based reward learning instead of reward engineering | Code generation policy initialized from CodeT5plus-large · APPS | Pass@50: 9.81 -> 10.19 | Helped |
| Use hierarchical preference-based reward learning instead of reward engineering | Code generation policy initialized from CodeT5plus-large · MBPP | Pass@1: 6.58 -> 7.40 | Helped |
| Use HERON-DPO rather than decaying-weight reward engineering | Phi-2 fine-tuned on HelpSteer · HelpSteer test | Average win rate vs SFT: 57.06 -> 64.57 | Helped |
| Use HERON rather than reward engineering | Robotic control policy · Ant | Ground-truth reward: 0.88 (0.02) -> 1.0 (0.01) | Helped |
| Use HERON rather than reward engineering | Robotic control policy · Hopper | Ground-truth reward: 0.72 (0.05) -> 0.78 (0.04) | Helped |
| Use HERON rather than reward engineering | Robotic control policy · Cheetah | Ground-truth reward: 0.61 (0.04) -> 0.62 (0.04) | Helped |
| Use HERON rather than reward engineering | Robotic control policy · Pendulum | Ground-truth reward: 0.99 (0.0) -> 1.0 (0.0) | No clear effect |
| Add post-training reward scaling with α (ablation) | HERON code-generation reward · APPS | Pass@K | No clear effect |
| Vary the preference-elicitation margin δ_i (ablation) | HERON · Experiments reported | Policy performance | No clear effect |
| Change the ranking of feedback signals (ablation) | HERON traffic-light policy · Traffic light control | Prioritized-signal performance | Helped |
| Place wait time lower in the feedback hierarchy (ablation) | HERON traffic-light policy · Traffic light control | Reward engineering reward | Hurt |
| Change the vehicle speed limit during training (ablation) | Reward-engineering policy · Traffic light control, changed-speed environment | Evaluation reward | Hurt |
Surprising findings
- Reward engineering with a tuned geometric weighting is sensitive to environment changes and often fails to beat HERON even when given the hierarchy.
- HERON performs only moderately (below ground-truth) on robotics tasks, showing its benefit is smaller when hierarchy is unclear.
Limitations
- Requires a strict importance ranking over feedback signals; equally important signals are not naturally handled.
- Benefit is smaller in robotics where the hierarchy is unclear, with performance still below ground-truth reward in several tasks.
- Hierarchy must be provided by a human overseer, and suitability is limited to settings with cheap, rankable feedback signals.
Experimental setup
Datasets: APPS MBPP HelpSteer SUMO traffic simulation (Flow framework) PyBullet robotics environments
Benchmarks: APPS MBPP HelpSteer PyBullet Ant/Hopper/HalfCheetah/Pendulum Traffic light control (SUMO/Flow)
Metrics: Pass@K filtered Pass@K win rate ground-truth reward t-test p-value
Baselines: Reward engineering (linear weights) CodeRL PPOCoder BC Ensemble policies Bayesian optimization reward tuning REINFORCE with decaying weights
Builds on: RLHF Bradley-Terry preference model DPO CodeRL PPO
Closest papers by meaning
- PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling (ICML 2024 · similarity 0.89)
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human Feedback (AAAI 2025 · similarity 0.88)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.87)
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental Design (NeurIPS 2025 · similarity 0.87)
- Listwise Reward Estimation for Offline Preference-based Reinforcement Learning (ICML 2024 · similarity 0.87)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.