AI Paper Atlas

Research map / Large language models / Alignment and preferences

From Curiosity to Caution: Mitigating Reward Hacking for Best-of-$N$ with Pessimism

Zhuohao Yu, Steven Wu, Adam Block

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add pessimism (caution) to the reward model scoreBest-of-N · GSM8Kfinal accuracy at N=512: 71.5 -> 81.1Improved
add pessimism (caution) to the reward model scoreBest-of-N · GSM8Kpeak accuracy: 79.3 -> 82.6Improved
add pessimism to the reward model scoreBest-of-N · GSM8Kdegradation: 7.7 -> 1.5Improved
use pessimism only instead of combining with RMBest-of-N · MATH-500peak accuracy: 12.3 -> 13.6Improved
add pessimism to the reward model scoreBest-of-N · MATH-500final accuracy at N=512: 8.5 -> 10.1Improved
use uncertainty penalty only instead of RM + pessimismBest-of-N · BBH-Hardfinal accuracy at N=512: 11.0 -> 22.1Improved
add pessimism to the reward model scoreBest-of-N · BBH-Harddegradation: 15.6 -> 7.5Improved
add reward-model score to pessimism-only selectionBest-of-N · MATH-500degradation: 1.7 -> 2.3Hurt
increase pessimism strength lambda from 0.0 to 0.8 (ablation)Caution · GSM8Kfinal accuracy at N=512: 71.2 -> 81.0Improved
use pessimism strength 1.0 (caution only) instead of 0.8 (ablation)Caution · GSM8Kfinal accuracy at N=512: 81.0 -> 79.8Hurt
distill reward model features instead of random network features (RND) (ablation)Caution · GSM8Kfinal accuracy at N=512 (lambda 0.8): 74.9 -> 81.0Improved
use a full-capacity predictor instead of lightweight predictor (ablation)Caution · GSM8Kfinal accuracy at N=512: 82.2 -> 80.2Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy