Research map / Large language models / Alignment and preferences
Transforming and Combining Rewards for Aligning Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the LSC-transformation (log-sigmoid of reward centered by a prompt-specific reference) for Bradley-Terry reward models and shows that summing transformed rewards approximates logical AND across properties, improving helpful+harmless RLHF and best-of-k.
Problem
Aligning LLMs from learned Bradley-Terry reward models raises two questions: which monotone transformation of the reward should be used for the alignment step, and how to combine multiple property reward models (e.g., helpfulness and harmlessness) without letting strong performance in one offset wea
Approach
Interpret alignment probabilistically as sampling conditioned on a response being "good" (preferred to a prompt-specific reference response). This yields utility u = log sigma(r - r_ref), and under conditional independence across properties, summing these transformed utilities corresponds to the log-probability of logical AND of goodness.
Key results
- In single-reward RLHF, the LSC-transformed reward gives better KL vs win-rate trade-offs than the raw reward for both helpfulness and harmlessness, under both PALM-2 and T5-XXL evaluators.
- For combined helpful+harmless alignment, summing log-sigmoid transformed rewards outperforms the weighted raw-reward sum, especially at high KL where the baseline reward-hacks.
- The transformed reward reduces shortcut behaviors such as list formatting in helpfulness and 'consult a professional' responses in harmlessness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSC-transformed reward instead of raw reward | PALM-2-XXS RLHF alignment · Anthropic Helpfulness | win rate | Helped |
| Use LSC-transformed reward instead of raw reward | PALM-2-XXS RLHF alignment · Anthropic Harmlessness | win rate | Helped |
| Sum LSC-transformed rewards instead of using a weighted sum of raw rewards | PALM-2-XXS best-of-k alignment · Anthropic Helpfulness and Harmlessness | win rate on both properties | Helped |
| Sum LSC-transformed rewards instead of using a weighted sum of raw rewards | PALM-2-XXS RLHF alignment · Anthropic Helpfulness and Harmlessness | win rate on both properties | Helped |
| Apply centering alone (ablation) | PALM-2-XXS RLHF alignment · Anthropic Helpfulness | RLHF performance | No clear effect |
| Apply log-sigmoid transformation alone, with a constant reference value of 0 (ablation) | PALM-2-XXS RLHF alignment · Anthropic Helpfulness | RLHF performance | Hurt |
Surprising findings
- Applying log-sigmoid alone without a prompt-specific reference (constant reference 0) hurts performance relative to the baseline.
- Centering alone does not change RLHF performance relative to the baseline.
Limitations
- Reference reward choice is a hyperparameter (85th quantile default; harmlessness uses canned responses) and requires sampling or estimating per-prompt quantiles.
- Aggregation correctness relies on the conditional independence assumption across properties.
- Evaluation relies on learned evaluators (T5-XXL, PALM-2) rather than human judgments; PPO-based experiments use a small PALM-2-XXS policy.
Experimental setup
Datasets: Anthropic Helpfulness dataset Anthropic Harmlessness dataset
Benchmarks: Anthropic Helpfulness and Harmlessness held-out prompts (win rate vs SFT quantiles)
Metrics: win rate vs SFT random sample win rate vs SFT quantile (T5-XXL proxy reward) win rate judged by PALM-2 zero-shot KL divergence to SFT
Baselines: raw Bradley-Terry reward utility weighted sum of raw rewards
Builds on: Bradley-Terry model RLHF with KL regularization PPO
Closest papers by meaning
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.93)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- Improving Context-Aware Preference Modeling for Language Models (NeurIPS 2024 · similarity 0.92)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.92)
- Data with High and Consistent Preference Difference Are Better for Reward Model (AAAI 2025 · similarity 0.91)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.