Research map / Large language models / Alignment and preferences

Bayesian Preference Learning for Test-Time Steerable Reward Models

NeurIPS 2026 · Large language models › Alignment and preferences · approach: Reward model design and preference models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Variational In-Context Reward Modeling (ICRM), which trains a classifier reward model to output a Beta posterior over Bradley-Terry preference probability conditioned on in-context preference demonstrations, enabling test-time steerability.

Problem

Classifier reward models are static after training and cannot adapt to new or mixed preference distributions at test time, and they are prone to over-optimization in RL.

Approach

Cast reward modeling as amortized variational inference: a KL-regularized Beta posterior over the BT preference probability, with a utility and a confidence head, conditioned on in-context preference demonstrations; the closed-form ELBO yields a global interior optimum and tempered preference means.

Key results

• On RM-Bench, ICRM average accuracy rises from 60.5 at N=1 to 70.8 at N=64, comparable to BTRM (70.2) and LLM-as-a-Judge (70.3).
• On moral dilemma calibration, ICRM ECE decreases from 0.099 at N=8 to 0.076 at N=64, lower than the Qwen3-4B LLM-as-a-Judge baseline.
• In RLVR, Qwen3-1.7B-Base trained with ICRM (lambda=0.1) reaches 53.8% MATH-500 accuracy versus 50.4% for the verifier and 51.4% for the BT reward model.

Surprising findings

• Reversed-label safety evaluation shows ICRM can be steered to prefer harmful responses, reaching up to 89% accuracy under reversed demonstrations.
• Policy trained with the learned ICRM reward outperforms the gold verifier on MATH-500 by about 3.4 points.

Limitations

• Context length limit of 16,384 tokens restricts the number of demonstrations; experiments stop at 32-64.
• Multi-objective Pareto dominance over static RMs depends on the backbone; Llama-3.2-3B does not uniformly dominate.
• Steerable models can be pushed toward harmful preferences, requiring safeguards.

Experimental setup

Datasets: Skywork-Preferences-v0.2 INTELLECT-MATH SafeRLHF HHH-Alignment RM-Bench PRINCIPLES moral dilemma pairs

Benchmarks: RM-Bench SafeRLHF HHH-Alignment MATH-500 moral dilemma preferences (PRINCIPLES)

Metrics: accuracy ECE RMSE hypervolume Pearson r R^2

Baselines: Bradley-Terry RM ArmoRM GRM LLM-as-a-Judge GRAM-r^2 Skywork-Reward-Llama-3.1-8B-v0.2 exact-match verifier

Builds on: Bradley-Terry model variational autoencoder ELBO GRPO Skywork-Reward

Closest papers by meaning

• Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.91)
• Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment (ICML 2024 · similarity 0.90)
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs (NeurIPS 2024 · similarity 0.90)
• RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
• AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization (ICML 2025 · similarity 0.90)
• Active Reward Modeling: Adaptive Preference Labeling for Large Language Model Alignment (ICML 2025 · similarity 0.90)