Research map / Large language models / Alignment and preferences
Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rewards-in-Context (RiC), which conditions a single foundation model on multiple normalized rewards in its prompt, trains it with SFT, and adapts to user preferences at inference via an analytically derived preference-to-reward mapping.
Problem
Multi-objective alignment of foundation models to heterogeneous, conflicting human preferences is costly because it requires training many RLHF policies (MORLHF) or many weight-interpolated policies (Rewarded Soups), and RL fine-tuning is unstable and expensive.
Approach
Relabel training prompts with multiple reward-model scores as text conditions and perform supervised fine-tuning of one model; at inference, map user preference weights to target reward values via a closed-form mapping derived from a regularized convex maximization, with online augmentation near the empirical Pareto front and multi-objective rejection sampling.
Key results
- On Helpful Assistant, RiC with two online iterations used about 153.2 GPU hours versus 1,477.1 for MORLHF (about 10.4%) and 622.7 for Rewarded Soups.
- RiC achieves a superior empirical Pareto front to MORLHF and Rewarded Soups on Helpful Assistant and Reddit Summary, with Reddit Summary gains attributed to retaining base-model faithfulness.
- On text-to-image with aesthetic and compressibility rewards, RiC adapts outputs to user preference weights, while Stable Diffusion v1.5 base cannot adapt.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RiC instead of MORLHF | Llama-2 7B multi-objective alignment · Helpful Assistant | empirical Pareto frontier | Helped |
| use RiC instead of Rewarded Soups | Llama-2 7B multi-objective alignment · Reddit Summary | empirical Pareto frontier | Helped |
| use RiC instead of MODPO | multi-objective alignment · Helpful Assistant and Reddit Summary | empirical Pareto frontier | Helped |
| add online training to offline RiC (ablation) | RiC · Helpful Assistant and Reddit Summary | empirical Pareto frontier | Helped |
| use p=2 preference-to-reward mapping instead of linear mapping (ablation) | RiC · Helpful Assistant and Reddit Summary | empirical Pareto frontier | Helped |
| use p=∞ preference-to-reward mapping instead of linear mapping (ablation) | RiC · Helpful Assistant and Reddit Summary | empirical Pareto frontier | Helped |
| replace p=2 mapping with p=∞ mapping (ablation) | RiC · Helpful Assistant and Reddit Summary | empirical Pareto frontier | No clear effect |
| add MORS sample rejection during online fine-tuning (ablation) | RiC · multi-objective alignment | empirical Pareto frontier | Helped |
| add online fine-tuning without MORS (ablation) | RiC · multi-objective alignment | empirical Pareto frontier | Helped |
| add original offline samples to the online fine-tuning buffer (ablation) | RiC · multi-objective alignment | empirical Pareto frontier | No clear effect |
| use multi-reward conditioning for positively correlated rewards (ablation) | RiC · Helpful Assistant and Reddit Summary | Pareto-front quality | Hurt |
| use RiC instead of MORLHF for GPU-hour cost | Llama-2 7B alignment with two objectives · Helpful Assistant | GPU hours: 10.4% of the GPU hours required by MORLHF | Helped |
Surprising findings
- SFT on multiple rewards degrades the 'faithful' reward on Reddit Summary, and RiC restores base-model faithfulness while improving preference rewards.
- RiC performs poorly when the two rewards are positively correlated, with no clear Pareto front.
Limitations
- Performs poorly when rewards are positively correlated, lacking a clear Pareto front.
- Relies on the approximate linear normalization of reward-to-objective maps and heuristic choice of p and lambda for p=infinity.
- Malicious users could set harmful desired rewards through the prompt conditioning.
Experimental setup
Datasets: Anthropic/hh-rlhf openai/summarize_from_feedback LAION-5B subset COCO karpathy test captions
Benchmarks: Anthropic/hh-rlhf test prompts (Helpful Assistant) Reddit summary test subset (2k prompts) COCO karpathy test captions (text-to-image)
Metrics: average normalized test reward per objective empirical Pareto front GPU hours
Baselines: MORLHF Rewarded Soups SFT Llama-2 base MODPO Stable Diffusion v1.5
Builds on: Rewarded Soups MORLHF reward conditional training PPO LoRA
Closest papers by meaning
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.90)
- Bayesian Preference Learning for Test-Time Steerable Reward Models (NeurIPS 2026 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.89)
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.89)
- MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.