Research map / Large language models / Alignment and preferences
GRAM: A Generative Foundation Reward Model for Reward Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRAM, a generative reward model pre-trained on unlabeled response pairs via response generation and then fine-tuned on preference labels with label smoothing, shown to generalize better than discriminative and generative baselines.
Problem
Reward models are usually trained discriminatively on labeled human preference data, which limits generalization and makes them costly to adapt; the paper asks how to exploit unlabeled data to train a foundation reward model.
Approach
Pre-train a generative LLM reward model by generating a pair of responses given an input using unlabeled data, then fine-tune it to output preference tokens with label smoothing, which the authors show equals a regularized Bradley-Terry loss.
Key results
- GRAM (LLaMA-3.1-8B-Instruct) reaches 85.1 average RewardBench accuracy vs 80.0 for the generative RM baseline and 74.1 for the discriminative RM, with 85.4 on HHH-Alignment.
- GRAM with LLaMA-3.2-3B-Instruct achieves 83.6 on RewardBench average, versus 78.8 for the generative baseline.
- After PPO fine-tuning on AlpacaEval, GRAM yields 12.63 win rate vs 6.75 for the generative baseline and 4.56 for SFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on unlabeled response pairs before supervised preference fine-tuning (ablation) | Generative reward model · RewardBench | accuracy: losing 3.9 points | Helped |
| Remove supervised preference fine-tuning and use only the unlabeled first stage (ablation) | GRAM · RewardBench and summarization adaptation | accuracy | No clear effect |
| Include both response-generation terms in first-stage pre-training (ablation) | GRAM · RewardBench and summarization adaptation | accuracy: 1.9 points | Helped |
| Use domain-specific summarization data for pre-training (ablation) | GRAM · Summarization adaptation | accuracy: 71.6 -> 74.7 | Helped |
| Exclude summarization-related data from pre-training (ablation) | GRAM · Summarization adaptation | accuracy: 71.6 -> 67.4 | Hurt |
| Add GRAM label smoothing (ablation) | Generative reward model · Unified-Feedback | accuracy: 68.4 -> 70.6 | Helped |
| Add GRAM label smoothing (ablation) | Generative reward model · RewardBench | accuracy: 82.1 -> 83.6 | Helped |
| Increase unlabeled pre-training data | GRAM · RewardBench | accuracy | Helped |
| Use GRAM instead of the generative reward model baseline | LLaMA-3.1-8B-Instruct reward modeling · RewardBench | average accuracy: 80.0 to 85.1 | Helped |
| Use GRAM rather than G-Label Smoothing as the PPO reward model | PPO fine-tuning · AlpacaEval2 | WinRate: 8.82 -> 12.63 | Helped |
| Use GRAM instead of the non-pre-trained Vanilla RM | Summarization adaptation · Summarization adaptation | accuracy: 56.5 -> 71.6 | Helped |
| Fine-tune GRAM with 3k summarization samples instead of the full 92k | GRAM · Summarization adaptation | accuracy: 75.6 vs. 77.8 | No clear effect |
Surprising findings
- Discriminative reward models are better on the in-distribution test set but generalize worse OOD than generative ones.
- Removing the second (supervised) stage, i.e. using only unlabeled pre-training, still gives comparable performance on pair-wise ranking and adaptation.
Limitations
- Underperforms discriminative models with label smoothing/regularization on the in-distribution UniFeed test set for some settings.
- Freezing and regularization baselines show model-dependent gains; gains are not consistent across model sizes.
- Pre-training scale (training budget) is much smaller than that of LLM foundation models; large-scale unlabeled data not explored beyond 400k.
Experimental setup
Datasets: Unified-Feedback RewardBench HHH-Alignment AlpacaEval2 AlpacaFarm TL;DR comparison data Alpaca
Benchmarks: UniFeed (Unified-Feedback test) RewardBench HHH-Alignment AlpacaEval2 best-of-n AlpacaEval (win rate after PPO)
Metrics: pairwise ranking accuracy average accuracy best-of-n proxy and oracle scores win rate length-controlled win rate
Baselines: GPT-4o GPT-4 GPT-3.5-turbo ArmoRM-Llama3-8B-v0.1 pair-preference-model-LLaMA3-8B GPM-Gemma-2B Selene-1-Mini-Llama-3.1-8B Skywork-Critic-Llama-3.1-8B Discriminative RM Generative RM
Builds on: Bradley-Terry reward modeling generative reward models (LLM-as-judge style) label smoothing instruction fine-tuning
Closest papers by meaning
- Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.92)
- Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.91)
- Learning Goal-Conditioned Representations for Language Reward Models (NeurIPS 2024 · similarity 0.91)
- Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.91)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.