Research map / Reinforcement learning / RL from feedback
Reward Learning from Multiple Feedback Types
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines six explicit human feedback types (ratings, comparisons, demonstrations, corrections, descriptions, descriptive preferences), simulates them from expert RL policies in ten environments, trains reward models per type, and tests a uncertainty-weighted ensemble for multi-type reward learning.
Problem
Preference-based feedback is the dominant form for reward learning, but learning from diverse feedback types and combining them has been little explored, and there is no consistent way to define and simulate multiple explicit feedback types.
Approach
Formalize six explicit feedback types under a common reward-rational-choice view, generate consistent simulated feedback for each from expert policies and a shared noise scheme (noise injected into the underlying reward), train one reward model per type, and combine them via averaging or inverse-uncertainty-weighted ensembling.
Key results
- Descriptive feedback and descriptive preferences generally yield the strongest downstream RL results, with demonstrative feedback outperforming others in Swimmer-v5.
- Comparative feedback is matched or surpassed by other feedback types across environments.
- The uncertainty-weighted ensemble matches the best single feedback type in HalfCheetah-v5 but shows limited gains in Walker2d-v5, performing at or below the single-type average there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use descriptive feedback or descriptive preferences instead of other single feedback types | reward learning and downstream RL · RL environments | downstream performance | Helped |
| use demonstrative feedback instead of other feedback types | reward learning and downstream RL · Swimmer-v5 | downstream performance | Helped |
| combine all six feedback-type reward models in an ensemble | learned reward models · HalfCheetah-v5 | episode return | Helped |
| combine all six feedback-type reward models in an ensemble | learned reward models · Walker2d-v5 | episode return | Mixed |
| introduce moderate noise into corrective feedback | corrective feedback reward learning · HalfCheetah-v5 | downstream RL performance | Helped |
| increase feedback noise | descriptive feedback · HalfCheetah-v5 and Walker-v5 | correlation with ground-truth reward | Helped |
| increase feedback noise | evaluative and comparative feedback · HalfCheetah-v5 and Walker-v5 | correlation with ground-truth reward | Hurt |
| use comparative feedback rather than other feedback types | reward learning and downstream RL · RL environments | downstream performance | Mixed |
Surprising findings
- Moderate noise sometimes improves downstream RL performance despite hurting reward learning, particularly for corrective feedback.
- Reward correlation with ground truth is weakly related to downstream RL performance; demonstrative feedback yields near-expert Humanoid performance despite low correlation.
Limitations
- Feedback is simulated from expert policies and additive Gaussian noise, not real human feedback.
- Expert policies may be suboptimal, and learned rewards are only well-defined over the sampled trajectory distribution.
- Joint ensemble gains are environment-specific and fail in some cases such as Walker2d-v5; the ensemble is only a proof of concept with pre-trained models.
Experimental setup
Datasets: simulated feedback generated from expert RL rollouts in Mujoco, MetaWorld sweep-into-v2, HighwayEnv
Benchmarks: Mujoco HalfCheetah-v5 Mujoco Walker2d-v5 Mujoco Swimmer-v5 Mujoco Ant-v5 Mujoco Hopper-v5 Mujoco Humanoid-v5 MetaWorld sweep-into-v2 HighwayEnv merge-v0 HighwayEnv highway-fast-v0 HighwayEnv roundabout-v0
Metrics: episode return normalized by expert return success rate validation loss correlation with ground-truth reward
Baselines: comparative (preference) feedback descriptive feedback behavioral cloning ground-truth reward
Builds on: PPO SAC Bradley-Terry model reward-rational choice Stable-Baselines3
Closest papers by meaning
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.91)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.91)
- Reward Modeling with Ordinal Feedback: Wisdom of the Crowd (ICML 2025 · similarity 0.90)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.90)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.89)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.