Research map / Large language models / Alignment and preferences
Preference Learning for AI Alignment: a Causal Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper framing LLM reward modelling from preference data as a causal problem, identifying assumptions (consistency, unconfoundedness, positivity, latent sufficiency/overlap) and showing via small experiments that user-objective confounding and limited latent overlap hurt BTL reward models, with a causally-inspired multihead+adversarial architecture helping.
Problem
Reward models trained from pairwise human preference data may fail to generalise to novel prompt-response pairs because of spurious correlations, preference heterogeneity, confounding by user-specific objectives, and limited latent overlap in observational preference data.
Approach
Frames preference learning within the potential outcomes framework, states identifiability assumptions for reward differences, and proposes a multi-objective reward architecture (Multihead) with an adversarial head that removes prompt-type information from the latent representation to mitigate confounding by user objectives.
Key results
- On UltraFeedback, OOD accuracy drops from 69.3% (rho=0.0) to 57.8% (rho=0.9), while ID accuracy stays near 67-69%.
- On HH-RLHF, at rho=1.0 all three reward models fall to just below 55% accuracy on inconsistent examples.
- The Multihead architecture significantly outperforms the Base model and the adversarial variant further improves accuracy, especially under strong confounding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase training correlation between truthfulness and instruction-following | BTL reward model · UltraFeedback OOD | accuracy: 64.5 ± 0.2 -> 57.8 ± 0.1 | Hurt |
| Increase training correlation between truthfulness and instruction-following | BTL reward model · UltraFeedback ID | accuracy: 69.3 ± 0.2 -> 67.6 ± 0.2 | Hurt |
| Use the Multihead architecture instead of the Base architecture | HH-RLHF multi-objective reward model · HH-RLHF inconsistent test samples | accuracy | Helped |
| Add adversarial balancing to the Multihead architecture | HH-RLHF multi-objective reward model · HH-RLHF inconsistent test samples | accuracy | Helped |
| Reduce alignment between user objective and prompt type in training data | HH-RLHF reward models · HH-RLHF inconsistent test samples | accuracy | Helped |
| Train with fully confounded data (ρ=1.0) | HH-RLHF reward models · HH-RLHF inconsistent test samples | accuracy: falling just below 55% | Hurt |
Surprising findings
- Even at moderate training correlation (rho=0.6) OOD accuracy drops to 59.7% versus 67.8% ID, showing that latent overlap violations matter even though identifiability holds in the infinite-data limit.
Limitations
- Assumptions are sufficient but not necessary for identifiability.
- Experiments assume user objectives are observed, not unobserved confounding in practice.
- Experiments use pre-computed frozen Llama-3-8B embeddings with small MLP reward heads, not full LLM fine-tuning.
Experimental setup
Datasets: UltraFeedback HH-RLHF (augmented with counterfactual objective labels)
Benchmarks: UltraFeedback (ID/OOD correlation splits) HH-RLHF augmented test split
Metrics: test-time preference accuracy
Baselines: Base concatenated-objective reward model Multihead reward model
Builds on: Bradley-Terry-Luce model potential outcomes framework gradient reversal RLHF reward modelling
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.93)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.92)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.92)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.91)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.91)
- Bayesian Preference Learning for Test-Time Steerable Reward Models (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.