Research map / Reinforcement learning / RL from feedback
Improving Reward Model Generalization from Adversarial Process Enhanced Preferences
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes APEC, which generates preference data for reward learning by pairing policies with widely separated iteration indices from a full adversarial imitation learning (DAC) run, improving preference coverage and reward-model generalization from suboptimal demonstrations.
Problem
Automated preference generation from suboptimal demonstrations often yields preference data with poor coverage of high-return regions, producing reward models that generalize poorly and cannot support policy learning.
Approach
Use policies saved at different iterations of an adversarial imitation learning run (later policies are preferred over earlier ones, with a theoretical value-gap bound shrinking with iteration index), select pairs with large index gaps, and filter pairs with a Wasserstein-distance criterion to get accurate, well-covered preferences.
Key results
- APEC reward correlation averages 0.72 on MuJoCo and 0.79 on DMControl, versus 0.05-0.47 for baselines (e.g., Ant 0.94, HalfCheetah 0.91).
- APEC return correlation averages 0.99 on MuJoCo and 0.96 on DMControl; preference accuracy averages 90% (MuJoCo) and 86% (DMControl).
- APEC achieves better-than-demonstration policy performance on 7 of 8 tasks, and generated-preference accuracy of 98%, versus 92% for SSRR and 80% for D-REX.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use APEC-generated preferences instead of baseline-generated preferences | reward model training · Mujoco tasks | reward correlation | Helped |
| Use APEC-generated preferences instead of baseline-generated preferences | reward model training · DMControl tasks | return correlation | Helped |
| Use APEC-generated preferences instead of baseline-generated preferences | reward model training · test trajectory pairs | preference accuracy | Helped |
| Add the Wasserstein distance criterion to preference generation (ablation) | APEC · Mujoco and DMControl tasks | reward correlation | Helped |
| Use fewer collection policies (ablation) | APEC · Mujoco and DMControl tasks | reward correlation | Hurt |
Surprising findings
- Baselines such as AIRL and LERP often fail to exceed demonstration performance, which the authors attribute to reward hacking rather than insufficient data.
Limitations
- Requires environment interaction to generate preferences, so it does not directly apply to offline settings.
- As a fully data-driven method, it may still suffer from reward hacking on more complex tasks.
- Theoretical bound relies on tabular-style quantities (|S||A|) and the Wasserstein criterion is a heuristic addition.
Experimental setup
Datasets: MuJoCo suboptimal demonstrations (SAC checkpoints at 50-80% of optimal) DMControl suboptimal demonstrations (DrQ-v2 checkpoints)
Benchmarks: MuJoCo (Ant, HalfCheetah, Hopper, Humanoid, Walker2d) DMControl (cheetah run, walker run, walker walk)
Metrics: reward correlation return correlation preference accuracy policy return (better-than-demo) projection distance (coverage) generation accuracy
Baselines: D-REX AIRL SSRR LERP
Builds on: D-REX DAC T-REX AIRL LERP
Closest papers by meaning
- A Minimaximalist Approach to Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.89)
- Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking (ICLR 2025 · similarity 0.89)
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.88)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.88)
- Preference Learning with Response Time: Robust Losses and Guarantees (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.