Research map / Large language models / Alignment and preferences
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes General Preference embedding Model (GPM), which embeds responses in a latent space with a skew-symmetric operator to model intransitive preferences at linear query cost, plus General Preference Optimization (GPO) for RLHF-style alignment.
Problem
Bradley-Terry reward models assume transitive scalar rewards and cannot express intransitive (cyclic) human preferences, while pairwise preference models need quadratic queries over K responses.
Approach
Represent each response as a vector in latent space and define the preference score as a skew-symmetric bilinear form with prompt-conditioned eigenvalue scaling; this generalizes BT (d=1) while keeping O(K) query cost, and GPO uses the averaged preference score as the optimization signal in place of win probability.
Key results
- On RewardBench with Gemma-2B-it, GPM (embedding dim 6) averages 82.29 vs 74.85 for the BT RM.
- On cyclic preference datasets, GPM reaches 100% accuracy while BT RM scores 50.0-62.9%.
- On AlpacaEval 2.0 with Llama3-8B-it, GPO with GPM at iteration 3 (2B) reaches 48.25 WR vs 44.20 for BT RM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BT RM with GPM | Gemma-2B-it preference modeling · Cyclic No. 1 | accuracy: 62.4 -> 100.0 (+37.6) | Helped |
| Replace BT RM with GPM | Gemma-2B-it preference modeling · Cyclic No. 2 | accuracy: 61.6 -> 100.0 (+38.4) | Helped |
| Replace BT RM with GPM | Gemma-2B-it preference modeling · Cyclic No. 3 | accuracy: 50.0 -> 100.0 (+50.0) | Helped |
| Replace BT RM with GPM | Gemma-2B-it preference modeling · Cyclic No. 4 | accuracy: 62.9 -> 100.0 (+37.1) | Helped |
| Replace BT RM with GPM, using embedding dimension 6 | Gemma-2B-it · RewardBench Average | score: 74.85 -> 82.29 (+7.44) | Helped |
| Replace BT RM with GPM, using embedding dimension 8 | Llama-3.1-8B-Instruct · RewardBench Average | score: 90.56 -> 91.90 (+1.34) | Helped |
| Use embedding dimension 6 instead of 4 (ablation) | GPM with Gemma-2B-it · RewardBench Average | score: 80.43 -> 82.29 | Helped |
| Use embedding dimension 8 instead of 6 (ablation) | GPM with Llama-3.1-8B-Instruct · RewardBench Average | score: 91.86 -> 91.90 | Helped |
| Use embedding dimension 2 instead of 8 (ablation) | GPM with Llama-3.1-8B-Instruct · RewardBench Chat-Hard | score: 87.50 -> 88.38 | Helped |
| Use GPM instead of BT RM | GPO with 2B reward model · AlpacaEval 2.0, iteration 1 | WR: 33.08 -> 37.95 (+4.87) | Helped |
| Use GPM instead of BT RM | GPO with 2B reward model · AlpacaEval 2.0, iteration 2 | WR: 39.90 -> 42.96 (+3.06) | Helped |
| Use GPM instead of BT RM | GPO with 8B reward model · AlpacaEval 2.0, iteration 3 | WR: 38.56 -> 41.54 (+2.98) | Helped |
Surprising findings
- With Llama-3.1-8B-Instruct, GPM gains on RewardBench are small (91.90 vs 90.56 average), and BT RM with dim 1 is best on Reasoning.
- On AlpacaEval 2.0 with the 8B base, GPM WR gains over BT RM are present but the LC win rate of GPM at some iterations is lower than BT RM.
Limitations
- Gains on RewardBench are inconsistent across base models and embedding dimensions.
- The policy-optimization convergence theorem relies on bounded scores and an approximation (log Z replaced by 0) used in practice.
- Cyclic preference dataset is small (216-363 instances per subset) and constructed from UltraFeedback ratings.
Experimental setup
Datasets: Skywork Reward Data Collection (decontaminated) UltraFeedback RewardBench CyclicPreference (constructed from UltraFeedback)
Benchmarks: RewardBench CyclicPreference AlpacaEval 2.0
Metrics: accuracy length-controlled win rate win rate average length
Baselines: Bradley-Terry reward model SPPO PairRM
Builds on: Bradley-Terry model SPPO DPO RLHF
Closest papers by meaning
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.93)
- Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis (NeurIPS 2025 · similarity 0.92)
- Online Preference Alignment for Language Models via Count-based Exploration (ICLR 2025 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.91)
- On-the-fly Preference Alignment via Principle-Guided Decoding (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.