Research map / Reinforcement learning / RL from feedback
Improving Generalization of Alignment with Human Preferences through Group Invariant Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes group invariant policy learning for RLHF: automatically infers data groups by maximizing return variance across groups, then adds a variance regularizer and an adaptive KL penalty, improving PPO stability and generalization on Anthropic-HH and Reddit TL;DR with Llama 2 7B.
Problem
RL-based RLHF exploits shortcuts and neglects challenging samples, leading to unstable training and poor generalization across data groups, and group annotations are costly or unavailable.
Approach
Jointly infer soft group labels via a classifier on the critic and adversarially maximize return variance across groups, then penalize that variance in the policy objective and use a group-dependent (adaptive) KL penalty that relaxes constraints on hard data.
Key results
- On Anthropic-Harmful, the method reaches 40.4% win / 25.9% lose against PPO w/ KL under GPT-4 judging, versus 40.4/25.9 for the proposed method in the ablation table; PPO w/ KL is the reference.
- Against PPO (no KL) on summarization, losing rate drops from 32.2% to 12.8% on OOD data, per the authors.
- Ablation: removing group invariant learning drops harmful win rate from 40.4% to 31.1%; removing dynamic KL drops it to 35.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add group invariant learning (ablation) | Ours (Llama 2 7B) · Anthropic-Harmful | win rate: 31.1 -> 40.4 | Helped |
| add group invariant learning (ablation) | Ours (Llama 2 7B) · Anthropic-Helpful | win rate: 23.4 -> 29.5 | Helped |
| add group invariant learning (ablation) | Ours (Llama 2 7B) · OpenAI-Summary | win rate: 29.3 -> 34.1 | Helped |
| use adaptive rather than fixed KL penalty (ablation) | Ours (Llama 2 7B) · Anthropic-Harmful | win rate: 35.2 -> 40.4 | Helped |
| use adaptive rather than fixed KL penalty (ablation) | Ours (Llama 2 7B) · Anthropic-Helpful | win rate: 24.6 -> 29.5 | Helped |
| use adaptive rather than fixed KL penalty (ablation) | Ours (Llama 2 7B) · OpenAI-Summary | win rate: 31.3 -> 34.1 | Helped |
Surprising findings
- Reward distribution under the proposed method is Gaussian-like, while PPO produces a long-tailed distribution on training data.
- Macro average loss performs worse than PPO in the authors' comparison, which they attribute to aggravated reward hacking.
Limitations
- Relies on GPT-4 and human win-rate evaluation rather than standard benchmarks; hyperparameter beta_policy sensitive above 0.1.
- Experiments limited to Llama 2 7B on two tasks with binary groups in the main results.
Experimental setup
Datasets: ShareGPT (52k SFT conversations) Anthropic-RLHF-HH Reddit TL;DR PKU-SafeRLHF (OOD) CNN/DailyMail (OOD)
Benchmarks: Anthropic-HH harmless Anthropic-HH helpful OpenAI Summary (TL;DR) PKU-SafeRLHF (OOD) CNN/DailyMail (OOD)
Metrics: win rate tie rate lose rate chi-squared test p-value
Baselines: SFT PPO PPO w/ KL DPO Macro average loss Group DRO JTT
Builds on: PPO Invariant Risk Minimization Group DRO Just Train Twice DPO
Closest papers by meaning
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.91)
- Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (ICLR 2024 · similarity 0.90)
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.90)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.90)
- SALMON: Self-Alignment with Instructable Reward Models (ICLR 2024 · similarity 0.89)
- Clone-Robust AI Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.