Research map / Reinforcement learning / RL from feedback
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Variational Preference Learning (VPL), a latent-variable, ELBO-trained reward model that infers user-specific latents from a few preference labels to handle divergent preferences in RLHF, evaluated on simulated control tasks and LLM reward modeling.
Problem
Standard RLHF fits a unimodal Bradley-Terry-Luce reward model that averages over divergent preferences across a diverse user population, yielding inaccurate rewards and poor performance for subgroups.
Approach
Model each annotator with a latent user context z inferred by a variational encoder from a few of their labeled comparisons, learn z-conditioned reward models and policies via an ELBO, and scale rewards using expected pairwise preference likelihoods (VPL-SPO) to avoid reward-scale variation across latents.
Key results
- On LLM reward modeling with GPT2, VPL reaches 100% on Pets (Divergent) versus 63.27% for BTL and 70.62% for DPL.
- On UF-P-2 with Llama2-7B, VPL reaches 76.41% accuracy versus 47.17% for BTL and 49.51% for DPL.
- VPL with active learning achieves comparable policy performance with roughly 2 queries versus about 8 random queries in simulated control.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VPL instead of BTL or DPL | GPT2 reward model · Pets (Divergent) | reward accuracy: 63.27 ± 0.57 -> 100 ± 0.00; 70.62 ± 1.13 -> 100 ± 0.00 | Helped |
| Use VPL instead of BTL or DPL | GPT2 reward model · Pets (Full) | reward accuracy: 94.92 ± 0.00 -> 100 ± 0.00; 95 ± 0.00 -> 100 ± 0.00 | Helped |
| Use VPL instead of BTL or DPL | GPT2 reward model · UF-P-2 | reward accuracy: 49.84 ± 0.14 -> 74.75 ± 2.01; 49.57 ± 0.42 -> 74.75 ± 2.01 | Helped |
| Use VPL instead of BTL or DPL | GPT2 reward model · UF-P-4 | reward accuracy: 53.48 ± 0.03 -> 61.49 ± 0.03; 52.92 ± 0.06 -> 61.49 ± 0.03 | Helped |
| Use VPL instead of BTL or DPL | Llama2-7B reward model · UF-P-2 | reward accuracy: 47.17 -> 76.41; 49.51 -> 76.41 | Helped |
| Use VPL rather than BTL on a unimodal preference dataset (ablation) | UF-P-4 reward model with one user's preferences · UF-P-4 unimodal setting | evaluation accuracy: 77.04% -> 77.16% | No clear effect |
| Scale rewards with VPL-SPO rather than batch normalization (ablation) | VPL policy · Maze-Navigation | success rate | Helped |
| Use batch-normalized rewards instead of VPL-SPO rewards (ablation) | VPL policy · Maze-Navigation | success rate | Hurt |
| Use VPL with max-norm rewards instead of unscaled VPL rewards (ablation) | VPL policy · Ravens-Manipulation | success rate | No clear effect |
| Select preference queries by maximum information gain instead of random queries (ablation) | VPL personalization · Simulated control tasks | policy performance at a given query count: same performance with only half the queries | Helped |
| Increase the number of context queries (ablation) | VPL user inference · Habitat-TidyBot | user identification accuracy | Helped |
| Use last-token embeddings instead of llm2vec or weighted pooling (ablation) | VPL LLM reward model · LLM preference datasets | reward model performance | Helped |
Surprising findings
- When 50% of context labels are flipped at test time, VPL performance falls to that of the BTL baseline.
- On a unimodal single-user dataset (UF-P-4 honest-only), VPL matches BTL (77.16% vs 77.04%).
Limitations
- Realistic preference datasets with diverse users do not exist at scale, so experiments rely on synthetic preferences.
- Assumes users can be queried on a fixed set of survey questions rather than naturally occurring conversation preferences.
- Performance depends on context length and on context label quality.
Experimental setup
Datasets: D4RL maze2d-medium-v2 Ravens benchmark Habitat UltraFeedback
Benchmarks: Maze-Navigation Ravens-Manipulation Habitat-Rearrange Habitat-Tidy Pets UltraFeedback-P-2 UltraFeedback-P-4
Metrics: task success rate reward prediction accuracy eval accuracy
Baselines: BTL reward model DPL (MeanVar) DPL (Categorical) Oracle goal-conditioned IQL
Builds on: Bradley-Terry-Luce model Distributional Preference Learning Implicit Q-Learning LoRA Self-Play Preference Optimization
Closest papers by meaning
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.93)
- Capturing Individual Human Preferences with Reward Features (NeurIPS 2025 · similarity 0.93)
- Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences (ICML 2024 · similarity 0.93)
- PILAF: Optimal Human Preference Sampling for Reward Modeling (ICML 2025 · similarity 0.92)
- Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback (ICLR 2024 · similarity 0.92)
- Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.