AI Paper Atlas

Research map / Large language models / Alignment and preferences

Why is Your Language Model a Poor Implicit Reward Model?

Noam Razin, Yong Lin, Jiarui Yao, Sanjeev Arora

ICLR 2026 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use implicit reward (log-prob ratio) instead of linear head on hidden statereward model · UltraFeedback in-distributionaccuracy: 0.752 -> 0.646Hurt
use implicit reward instead of explicit headreward model · UltraFeedback token-level shiftsaccuracy: 0.665 -> 0.602Hurt
use implicit reward instead of explicit headreward model · UltraFeedback-trained, domain shiftaccuracy: 0.621 -> 0.720Improved
use implicit reward instead of explicit headreward model · RewardMATH-trained, token-level shift (RewardBench math)accuracy: 0.988 -> 0.515Hurt
use implicit reward instead of explicit headreward model · RewardMATH-trained, in-distributionaccuracy: 0.971 vs 0.972No clear effect
use implicit reward instead of explicit headreward model · Persona paraphrased responsesaccuracyHurt
use implicit reward instead of explicit headreward model · UltraFeedback-trainednormalized absolute reward margin: 1.014 -> 0.813 in-distributionHurt
use implicit reward on Hamiltonian cycle verification (generation harder than verification)Pythia-1B reward model · Hamiltonian cycle verificationaccuracyNo clear effect
train EX-RM on all intermediate-token hidden states, and IM-RM without reference distribution (ablation)reward model · UltraFeedback generalization gapaccuracy gapNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy