Research map / Reinforcement learning / RL from feedback

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

ICML 2025 · Reinforcement learning › RL from feedback · approach: Theory of RLHF · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies an energy loss phenomenon in the final layer of LLMs during RLHF linked to reward hacking, proves a contextual-relevance bound, and proposes EPPO, a PPO variant that penalizes final-layer energy loss increase in the reward.

Problem

Reward hacking in RLHF, where policy optimization under a proxy reward model diverges from true human preferences and degrades contextual relevance; existing mitigations constrain the output space (KL, length penalties), which restricts policy optimization.

Approach

Measures energy loss (difference in L1 norms of a layer's input and output hidden states) in the LLM's final layer, shows it rises during RL with excessive rises marking hacking, and penalizes its deviation from the SFT model within the PPO reward, which the authors interpret as entropy regularization.

Key results

• EPPO outperforms PPO w/ KL and PPO w/ LP on Llama3-8B across four evaluation sets under GPT-4 judging (e.g., Anthropic-Helpful win/tie/lose 76/17/7 for PPO vs. 59/30/11 for PPO w/ KL, with EPPO comparisons reported in the paper's tables).
• EPPO reduces hacking-sample outliers in InfoRM latent space and keeps GPT-4 win rates from declining late in RL, where PPO deteriorates.
• EPPO combined with ODIN or InfoRM further improves win rates over those reward-modeling baselines alone.

Measured effects reported in the paper

Surprising findings

• Energy loss is reported to be empirically equivalent to its increase during RL, so the penalty uses an absolute deviation from the SFT model.
• The theoretical bound on contextual mutual information only shows suppression when energy loss is below the global energy offset; above that threshold the effect is stated to be an open question.

Limitations

• Theoretical bound holds only when energy loss is below the global energy offset; the regime above threshold is unresolved.
• Hacking identification relies on GPT-4 and InfoRM, and hacking samples in summarization were hard to identify with GPT-4, so InfoRM was used there.
• Hyperparameter eta trades off against reward; sensitivity is only in appendix.

Experimental setup

Datasets: ShareGPT Anthropic-HH Reddit TL;DR AlpacaFarm

Benchmarks: Anthropic-Helpful Anthropic-Harmless AlpacaFarm Reddit TL;DR

Metrics: win rate tie rate lose rate energy loss contextual dependency strength

Baselines: SFT PPO PPO w/ KL PPO w/ LP ERM-Mean ERM-WCO ERM-UWO WARM ODIN InfoRM

Builds on: PPO InfoRM ODIN contextual dependency strength

Closest papers by meaning

• InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling (NeurIPS 2024 · similarity 0.90)
• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.90)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking (ICLR 2025 · similarity 0.90)
• ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.89)
• WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.89)
• Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms (NeurIPS 2024 · similarity 0.88)