Research map / Reinforcement learning / RL from feedback
Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
Johannes Ackermann, Michael Noukhovitch, Takashi Ishida, Masashi Sugiyama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Reference Resets of the KL reference to GRPO (ablation) | GRPO with KL penalty · TL;DR win rate, Pythia 1B | win rate: 62.2% -> 78.1% | Improved |
| add Reference Resets to GRPO (ablation) | GRPO with KL penalty · TL;DR win rate, Qwen2.5-1.5B | win rate: 82.5% -> 82.0% | No clear effect |
| add Reference Resets to DPO (TR-DPO) | DPO · TL;DR win rate | win rate: 45.9% -> 45.5% (Pythia 1B) | No clear effect |
| lower KL penalty strength beta instead of using Reference Resets (ablation) | GRPO with KL penalty · TL;DR, Pythia 1B | gold model score | Hurt |
| replace KL penalty with explicit gradient regularization (ablation) | GRPO · AlpacaFarm, Qwen2.5-3B | win rate: 52.8% -> 59.2% | Improved |
| add KL penalty to unregularized RLHF (ablation) | GRPO · AlpacaFarm, Qwen2.5-3B | win rate: 44.2% -> 52.8% | Improved |
| use GR instead of Reference Resets with the strongest RM | GRPO · AlpacaFarm, Qwen2.5-1.5B, strongest RM | win rate | Hurt |
| add GR to GRPO with rule-based reward (ablation) | GRPO · GSM8K, Qwen2.5-0.5B | accuracy: 52.1 ± 0.3% -> 56.2 ± 1.0% | Improved |
| add KL penalty to GRPO with rule-based reward (ablation) | GRPO · GSM8K, Qwen2.5-0.5B | accuracy: 52.1 ± 0.3% -> 44.2 ± 1.2% | Hurt |
| add GR to GRPO with LLM-judge reward (ablation) | GRPO · GSM8K, Qwen2.5-1.5B | accuracy: 1.9% -> 67.8% | Improved |
| add GR to GRPO with LLM-judge reward (ablation) | GRPO · GSM8K, Qwen2.5-0.5B | accuracy: 20.7 ± 3.8% -> 41.0 ± 2.8% | Improved |
| add GR to GRPO with rule-based reward (ablation) | GRPO · MATH, Qwen2.5-1.5B-Instruct | pass@1: 54.8% -> 57.6% | Improved |
Closest papers by meaning
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
- Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.91)
- Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization (ICLR 2025 · similarity 0.91)
- The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.91)
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
