Research map / Large language models / Alignment and preferences
Improving Context-Aware Preference Modeling for Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes context-aware preference modeling (select a context, then evaluate context-specific preference), introduces RPR preference-reversal datasets, and finetunes a context-aware Mistral reward model that beats larger models on context-specific preference.
Problem
Pairwise preference labels are ambiguous and inconsistent because natural-language prompts underspecify the context (user identity, goals, criteria), and current reward models fail to make good use of added context.
Approach
Decompose reward modeling error into context inference error and context-specific prediction error via an upper bound, and train reward models conditioned on explicit context using preference-reversal datasets where the preferred response flips under an alternative context.
Key results
- Mistral CARM (7B) reaches 0.985 agreement with context on RPR Criteria vs 0.511 without context, outperforming Llama 3 Instruct 70B (0.925) and GPT-4 Turbo (0.901).
- Current models often ignore adversarial context: on Chatbot Arena with the Nonsense criteria, most models stay far from the 0.50 optimum, e.g., UltraRM 0.718.
- Conditioning on profiles inferred from 32 samples recovers most of the ground-truth context benefit (mean 0.762 vs 0.803 GT, vs 0.528 no context).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Finetune Mistral RM on RPR Criteria, RPR Scenarios, and UltraFeedback to create Mistral CARM | Mistral RM (7B) · RPR Criteria (CTX) | accuracy: 0.867 -> 0.985 | Helped |
| Finetune Mistral RM on RPR Criteria, RPR Scenarios, and UltraFeedback to create Mistral CARM | Mistral RM (7B) · RPR Scenarios (CTX) | accuracy: 0.749 -> 0.962 | Helped |
| Add context to preference evaluation | Preference models · RPR Criteria | accuracy: 0.523 -> 0.876 (UltraRM); 0.510 -> 0.867 (Mistral RM) | Helped |
| Add context to preference evaluation | Llama 3 (8B) · Preference Bench | accuracy: 0.831 -> 0.827 | Hurt |
| Add context to preference evaluation | Preference models · HHH Alignment | accuracy: 0.891 -> 0.905 (Mistral RM); 0.887 -> 0.919 (Mistral CARM) | Helped |
| Add context to preference evaluation | GPT-4 Turbo · Reward Bench | accuracy: 0.824 -> 0.821 | Hurt |
| Add context to preference evaluation | Preference models · RPR Scenarios | accuracy: 0.523 -> 0.773 (UltraRM); 0.510 -> 0.749 (Mistral RM) | Helped |
| Infer a user profile from preference examples and condition Mistral CARM on it | Mistral CARM · RPR profiles, mean (2 samples vs no context) | accuracy: 0.528 -> 0.637 | Helped |
| Infer a user profile from 32 preference examples and condition Mistral CARM on it | Mistral CARM · RPR profiles, mean (32 inferred samples vs ground-truth profile) | accuracy: 0.762 -> 0.803 | Hurt |
Surprising findings
- Models sometimes completely ignore added context and return their unconditioned prediction, even when the context is strongly phrased and adversarial.
- Llama 3 Instruct (70B) and the finetuned CARM outperform GPT-4 Turbo at interpreting GPT-4's own profile-conditioned preference labels.
Limitations
- Datasets are synthetic rather than collected from real human feedback, limiting practical relevance.
- Datasets are primarily English and limited to single-turn conversations.
- Context-aware models still have significant room for improvement on adversarial contexts.
Experimental setup
Datasets: RPR Criteria RPR Scenarios UltraFeedback Preference Bench Multifaceted Bench HHH Reward Bench Chatbot Arena
Benchmarks: RPR Criteria RPR Scenarios Preference Bench Multifaceted Bench HHH Reward Bench Chatbot Arena
Metrics: agreement accuracy
Baselines: UltraRM Prometheus-2 Mistral RM Llama 3 Instruct GPT-4 Turbo Llama 3
Builds on: Bradley-Terry model UltraFeedback Prometheus-2
Closest papers by meaning
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.92)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- Eliciting Human Preferences with Language Models (ICLR 2025 · similarity 0.91)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.90)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.90)
- SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.