Research map / Large language models / Alignment and preferences
Why is Your Language Model a Poor Implicit Reward Model?
Noam Razin, Yong Lin, Jiarui Yao, Sanjeev Arora
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use implicit reward (log-prob ratio) instead of linear head on hidden state | reward model · UltraFeedback in-distribution | accuracy: 0.752 -> 0.646 | Hurt |
| use implicit reward instead of explicit head | reward model · UltraFeedback token-level shifts | accuracy: 0.665 -> 0.602 | Hurt |
| use implicit reward instead of explicit head | reward model · UltraFeedback-trained, domain shift | accuracy: 0.621 -> 0.720 | Improved |
| use implicit reward instead of explicit head | reward model · RewardMATH-trained, token-level shift (RewardBench math) | accuracy: 0.988 -> 0.515 | Hurt |
| use implicit reward instead of explicit head | reward model · RewardMATH-trained, in-distribution | accuracy: 0.971 vs 0.972 | No clear effect |
| use implicit reward instead of explicit head | reward model · Persona paraphrased responses | accuracy | Hurt |
| use implicit reward instead of explicit head | reward model · UltraFeedback-trained | normalized absolute reward margin: 1.014 -> 0.813 in-distribution | Hurt |
| use implicit reward on Hamiltonian cycle verification (generation harder than verification) | Pythia-1B reward model · Hamiltonian cycle verification | accuracy | No clear effect |
| train EX-RM on all intermediate-token hidden states, and IM-RM without reference distribution (ablation) | reward model · UltraFeedback generalization gap | accuracy gap | No clear effect |
Closest papers by meaning
- Interpreting Language Reward Models via Contrastive Explanations (ICLR 2025 · similarity 0.91)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.89)
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.89)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
