Research map / Large language models / Alignment and preferences
Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fisher-information-based (D-optimal) active selection of preference pairs for Bradley-Terry reward modeling, applied to last-layer embeddings of LLM reward models, and benchmarks 8 selection strategies on Anthropic HH with Gemma-2B/7B and LLaMA3-8B.
Problem
Selecting the most informative preference pairs to annotate for reward model training is costly and under-explored; online annotation prioritization has mostly relied on heuristics.
Approach
Adapts classical experimental design (D-optimality on the Fisher information of the final linear layer) to choose preference comparisons that balance embedding-space exploration and informative mid-range reward differences, with a sensitivity-based gradient approximation to make the combinatorial selection tractable.
Key results
- D-opt and past-aware D-opt outperform Random, Entropy, Maxdiff, BatchBALD, Coreset and det(XtX) across Gemma-2B, Gemma-7B and LLaMA3-8B on Harmless with 5 seeds, with lower variance.
- Cross-prompt annotation improves efficiency across all methods, with entropy showing the largest gain.
- Authors estimate their method reaches similar reward-model performance with about 1,000 informative comparisons versus 20,000 random, a claimed 20x annotation cost reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D-opt on last-layer features for comparison selection | Active reward modeling · Harmless | 1−Spearman correlation | Helped |
| Use Past-Aware D-opt on last-layer features for comparison selection | Active reward modeling · Harmless | 1−Spearman correlation | Helped |
| Use D-opt or Past-Aware D-opt rather than alternative selection methods | Active reward modeling · Harmless | Best-of-N reward | Helped |
| Use D-opt or Past-Aware D-opt rather than alternative selection methods | Active reward modeling · Helpful | 1−Spearman correlation | Helped |
| Use D-opt or Past-Aware D-opt rather than alternative selection methods | Active reward modeling · Cross-prompt Harmless | 1−Spearman correlation | Helped |
| Use cross-prompt comparisons instead of in-prompt comparisons (ablation) | Active reward modeling · Harmless | 1−Spearman correlation | Helped |
| Use cross-prompt comparisons instead of in-prompt comparisons (ablation) | Active reward modeling · Harmless | Best-of-N reward | Helped |
| Use smaller annotation batches with more frequent retraining (ablation) | D-opt and Past-Aware D-opt · Harmless and Helpful | 1−Spearman correlation | Helped |
| Vary the hidden dimension of the reward-model MLP (ablation) | Active reward modeling · Hyper-parameter sensitivity analysis | Performance | No clear effect |
Surprising findings
- Entropy-based selection shows the largest efficiency gain when switching from in-prompt to cross-prompt comparisons, though D-opt remains best.
Limitations
- Relies on a well-trained embedding model; last-layer features must be informative.
- Uses open-source golden reward models as surrogate annotators rather than real human preferences.
- Theory is for linear Bradley-Terry; nonlinear MLP extension is a heuristic.
Experimental setup
Datasets: Anthropic Harmless (HH) Anthropic Helpful (HH)
Benchmarks: Anthropic Harmless Anthropic Helpful
Metrics: 1-Spearman correlation best-of-N reward (N=500)
Baselines: Random Entropy Maxdiff BatchBALD Coreset det(XtX) PA D-opt
Builds on: Bradley-Terry model BatchBALD Coreset (Huggins et al., 2016) Maximum difference / entropy sampling (Muldrew et al., 2024) Sener & Savarese core-set active learning
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Tool-Augmented Reward Modeling (ICLR 2024 · similarity 0.91)
- GRAM: A Generative Foundation Reward Model for Reward Generalization (ICML 2025 · similarity 0.91)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.91)
- Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment (ICML 2025 · similarity 0.91)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.