Research map / Large language models / Alignment and preferences
InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoRM, a variational information-bottleneck reward model that filters preference-irrelevant features to mitigate reward overoptimization in RLHF, plus a CSI indicator based on latent-space outliers for detecting overoptimization.
Problem
Reward overoptimization (reward hacking) in RLHF, caused by reward misgeneralization where reward models rely on spurious, preference-irrelevant features such as length.
Approach
Train the reward model with a variational information bottleneck objective so the latent representation keeps preference-relevant information and discards irrelevant information; outliers in this IB latent space correlate with overoptimized samples, which motivates the Cluster Separation Index (CSI) for detecting overoptimization and enabling early stopping.
Key results
- In simulated RLHF with 1.4B proxy RM and 25% label noise, InfoRM keeps gold and proxy scores rising while Standard RM's gold score declines (overoptimization).
- On Anthropic-Helpful under GPT-4 evaluation, InfoRM vs Standard RM achieves 54.5% win / 33.5% tie / 12.0% lose.
- CSI rises sharply between 600-700 training steps for Standard RM, while InfoRM keeps much lower CSI values, indicating CSI detects overoptimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InfoRM instead of Standard RM | RLHF reward modeling · Anthropic-Helpful | win rate: 54.5 | Helped |
| Use InfoRM instead of Standard RM | RLHF reward modeling · Anthropic-Harmless | win rate: 54.2 | Helped |
| Use InfoRM instead of Standard RM | RLHF reward modeling · AlpacaFarm | win rate: 45.1 | Helped |
| Use InfoRM instead of Standard RM | RLHF reward modeling · TL;DR Summary | win rate: 70.4 | Helped |
| Use InfoRM instead of Standard RM w/ KL | RLHF reward modeling · Anthropic-Helpful | win rate: 49.0 | Helped |
| Use InfoRM instead of Standard RM w/ KL | RLHF reward modeling · Anthropic-Harmless | win rate: 44.3 | Helped |
| Use InfoRM instead of Standard RM w/ KL | RLHF reward modeling · AlpacaFarm | win rate: 38.5 | Helped |
| Use InfoRM instead of Standard RM w/ KL | RLHF reward modeling · TL;DR Summary | win rate: 68.6 | Helped |
| Combine InfoRM with Ensemble RM | RLHF reward modeling · Anthropic-Harmless | win rate: 52.5 | Helped |
| Combine InfoRM with WARM | RLHF reward modeling · Anthropic-Harmless | win rate: 67.9 | Helped |
| Use InfoRM rather than Standard RM as the reward model | simulated RLHF with 1.4B proxy RM · AlpacaFarm and Flan | gold reward | Helped |
| Add KL penalty to Standard RM | simulated RLHF with 1.4B proxy RM and 25% label noise · AlpacaFarm | RLHF performance: 0.001 | Mixed |
Surprising findings
- Standard RM w/ KL performance degrades when KL penalty exceeds 0.001 in simulation, suggesting KL penalties can restrict policy optimization.
- Outliers in standard RM latent spaces do not reliably indicate overoptimized samples, so CSI is incompatible with standard RMs.
Limitations
- Evaluated only up to 7B parameter models; scaling to much larger models is unexplored.
- The overoptimization monitoring requires inference on test datasets and has latency; real-time lightweight metrics are needed.
- GPT-4 win rates are sensitive to prompt structure.
Experimental setup
Datasets: AlpacaFarm Anthropic-RLHF-HH Reddit TL;DR Flan FalseQA HelpSteer Mkqa Oasst1 OpenOrca Piqa PKU-SafeRLHF ShareGPT
Benchmarks: AlpacaFarm validation (self-instruct, Vicuna, Koala) Anthropic-RLHF-HH test Reddit TL;DR test
Metrics: gold reward win/tie/lose ratio CSI average response length
Baselines: Standard RM Standard RM w/ KL Ensemble RM WARM SFT model
Builds on: Information bottleneck Variational information bottleneck PPO Bradley-Terry model WARM Ensemble RM (UWO)
Closest papers by meaning
- The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking (ICML 2025 · similarity 0.90)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
- Information-Theoretic Reward Decomposition for Generalizable RLHF (NeurIPS 2025 · similarity 0.89)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.89)
- Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.89)
- Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.