Research map / Large language models / Alignment and preferences
Inference-Time Reward Hacking in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes reward hacking in inference-time alignment (BoN, SBoN) as inevitable under a broad class of mechanisms, proposes Best-of-Poisson (BoP) as a near-optimal reward-KL tilting approximation, and HedgeTune to pick the hacking-threshold parameter.
Problem
Inference-time alignment methods such as Best-of-n optimize an imperfect proxy reward, so true reward can first rise and then collapse (reward hacking); the problem is how to choose the inference-time parameter to avoid overoptimization.
Approach
Shows via variation-diminishing (TP2) arguments that the true reward as a function of inference-time strength has at most one interior extremum, introduces Best-of-Poisson (random Poisson sample count BoN) whose density approximates the optimal reward-KL tilt with KL gap about 1e-4 for uniform rewards, and tunes the parameter by root-finding on a residual (HedgeTune) using a one-time calibration a
Key results
- BoP approximates the optimal reward-KL tilted distribution with a KL gap bounded between 0 and 8e-4 for uniform rewards.
- HedgeTune recovers the best operating point for BoN and BoP on PPE verifiable setups, including BoN hacking on GPQA with Skywork-Llama-3.1 8B as proxy.
- In human-preference setups with Pythia 1.4B and AlpacaRM gold reward, hedging via lambda in SBoN or n in BoN/BoP mitigates hacking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Tune BoN sample count to the hacking threshold with HedgeTune | BoN · MMLU Pro, MATH, and GPQA | accuracy | Helped |
| Tune the Poisson rate to the hacking threshold with HedgeTune | Best-of-Poisson · MMLU Pro, MATH, and GPQA | accuracy | Helped |
| Tune the inverse temperature to hedge against reward hacking | Soft Best-of-n · MMLU Pro, MATH, and GPQA | reward-distortion tradeoff | Helped |
| Increase Best-of-n sample count beyond the hacking threshold | Best-of-n · GPQA | true reward | Hurt |
| Hedge inference-time sampling using its parameter (n for BoN and BoP, or λ for SBoN) | BoN, SBoN, and Best-of-Poisson · Human-preference setup, AlpacaFarm validation prompts | true reward | Helped |
| Use Best-of-Poisson instead of the reward-matched optimal tilted distribution | Best-of-Poisson · Uniform proxy rewards, matched expected reward | KL gap: 0 \le ... \le 8 \times10^{-4} | No clear effect |
Surprising findings
- If the proxy is always at odds with the true reward, the optimal hedge is the reference distribution itself (lambda=0).
Experimental setup
Datasets: PPE (Preference Proxy Evaluations) tlc4418/gold_labelled_gens AlpacaFarm
Benchmarks: MMLU Pro MATH GPQA AlpacaFarm
Metrics: expected true reward (accuracy) KL divergence to reference reward-distortion tradeoff
Baselines: Best-of-n Soft Best-of-n RLHF DPO
Builds on: Best-of-n sampling Soft Best-of-n RLHF KL-regularized objective Coste et al. reward model overoptimization
Closest papers by meaning
- Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment (ICML 2025 · similarity 0.92)
- Theoretical Limits of Language Model Alignment (NeurIPS 2026 · similarity 0.91)
- Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling (ICLR 2026 · similarity 0.90)
- On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference (ICML 2026 · similarity 0.90)
- WARM: On the Benefits of Weight Averaged Reward Models (ICML 2024 · similarity 0.89)
- Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.