Research map / Reinforcement learning / RL from feedback
Alleviating Shifted Distribution in Human Preference Alignment through Meta-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaRM, which uses an alternating meta-learning update to adapt a reward model to the shifted policy output distribution during iterative RLHF, without collecting new preference pairs.
Problem
Reward models lose their ability to distinguish responses from a policy whose output distribution shifts during RL training, and generalize poorly to out-of-distribution prompts; collecting new preference data each round is costly.
Approach
A meta-process takes a gradient ascent step on a response-difference loss computed on responses sampled from the shifted policy, then a vanilla preference loss on original pairs is minimized at the adapted parameters (MAML-style alternating update), so the RM stays discriminative on new-distribution responses while still fitting original preferences.
Key results
- MetaRM over rounds vs SFT on Anthropic-Harmless reaches 69% win / 3% lose under GPT-4 judging in round 3, with comparable human results (72% win).
- On summarization, round 4 achieves 78% win vs SFT under GPT-4 evaluation, versus 62% win for vanilla PPO against SFT.
- RM validation accuracy curves are preserved under MetaRM relative to vanilla RM training, while reward-difference distributions are more dispersed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iteratively adapt the reward model to shifted policy outputs using MetaRM | Llama-2-7B alignment on Anthropic-Harmless · Anthropic-Harmless, rounds 1–3 vs SFT | win rate: 44, 65, 69 (GPT-4); 48, 63, 72 (human) | Helped |
| Iteratively adapt the reward model to shifted policy outputs using MetaRM | Llama-2-7B alignment on Anthropic-Helpful · Anthropic-Helpful, rounds 1–3 vs SFT | win rate: 39, 62, 73 (GPT-4); 44, 65, 69 (human) | Helped |
| Iteratively adapt the reward model to shifted policy outputs using MetaRM | Llama-2-7B alignment on summarization · Summary, rounds 1–4 vs SFT | win rate: 51, 55, 67, 78 (GPT-4); 54, 57, 63, 77 (human) | Helped |
| Continue iterative MetaRM alignment beyond the best reported round | Llama-2-7B alignment · Anthropic-Harmless, Round 4 | win rate (GPT-4): 69 -> 64 | Hurt |
| Continue iterative MetaRM alignment beyond the best reported round | Llama-2-7B alignment · Anthropic-Helpful, Round 4 | win rate (GPT-4): 73 -> 67 | Hurt |
| Continue iterative MetaRM alignment beyond the best reported round | Llama-2-7B alignment · Summary, Round 5 | win rate (GPT-4): 78 -> 72 | Hurt |
| Use MetaRM rather than vanilla PPO | Llama-2-7B alignment · Anthropic-Harmless, comparison against Vanilla PPO | win rate: 54 win, 31 tie, 15 lose | Mixed |
| Vary MetaRM meta-process learning rate η (ablation) | Llama-2-7B alignment · Anthropic-Harmless, Anthropic-Helpful, Summary | win rate | No clear effect |
| Use MetaRM rather than other alignment baselines | Llama-2-7B OOD alignment · Out-of-distribution task | win rate: 39% (Helpful PPO); 40% (Harmless PPO); 48% (Helpful SFT); 52% (Harmless SFT) | Helped |
| Use MetaRM rather than vanilla PPO for response generation | Llama-2-7B alignment · Anthropic-Helpful and Summary | response diversity: 0.29 -> 0.25; 0.36 -> 0.33 | Helped |
Surprising findings
- Vanilla PPO and DPO reduce response diversity relative to SFT, while MetaRM slightly increases diversity on helpfulness and summarization.
Limitations
- Improvements saturate or decline after a few rounds (round 4 for dialogue, round 5 for summarization), indicating an upper limit.
- Evaluation relies mainly on GPT-4 win rates on only 100 prompts per comparison, with human evaluation on a subset.
Experimental setup
Datasets: Anthropic HH-RLHF ShareGPT Reddit TL;DR OpenAI summarization preference data Oasst1 PKU-SafeRLHF
Benchmarks: Anthropic HH-RLHF test Reddit TL;DR summarization PKU-SafeRLHF OOD Oasst1 OOD
Metrics: win rate tie rate lose rate SelfBLEU sentence-embedding diversity RM validation accuracy
Baselines: SFT Vanilla PPO DPO
Builds on: RLHF PPO DPO MAML
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.90)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.90)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.90)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.90)
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.