Research map / Large language models / Alignment and preferences
SALMON: Self-Alignment with Instructable Reward Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SALMON, an instructable reward model trained on synthetic principle-conditioned preferences that steers RL of a LLaMA-2-70b base model, yielding the Dromedary-2 assistant with only 6 ICL exemplars and 31 principles.
Problem
Aligning LLMs with minimal human supervision: current SFT+RLHF pipelines depend on costly human demonstrations and online in-distribution preferences needed to counter reward hacking.
Approach
An instructable reward model scores responses conditioned on arbitrary natural-language judging principles (including negated, prohibition-style ones), so reward hacking can be countered at RL time by editing principles instead of collecting fresh human preferences.
Key results
- Dromedary-2-70b reaches MT-Bench 7.4 and AlpacaEval win rate 88.32% using 6 annotations, versus LLaMA-2-Chat-70b at 6.9 and 92.66% with 1.4M preferences.
- Dromedary-2-70b scores 40.6 HumanEval P@1 vs 35.0 for LLaMA-2-Chat-70b and 66.3 BBH CoT vs 52.2, and 0.98 truthful on TruthfulQA.
- On Anthropic Adversarial reward evaluation, the Adversarial RM with anti-self-praise principles reaches 89.8% vs 56.3% for the PMP RM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Self-Align plus SALMON instead of LLaMA-2-Chat-70b alignment | LLaMA-2-70b · MT-Bench | score: 6.9 -> 7.4 | Helped |
| Apply Self-Align plus SALMON instead of LLaMA-2-Chat-70b alignment | LLaMA-2-70b · BBH Direct | score: 43.1 -> 51.4 | Hurt |
| Apply Self-Align plus SALMON instead of LLaMA-2-Chat-70b alignment | LLaMA-2-70b · BBH CoT | score: 52.2 -> 66.3 | Helped |
| Apply Self-Align plus SALMON instead of LLaMA-2-Chat-70b alignment | LLaMA-2-70b · HumanEval | P@1: 35.0 -> 40.6 | Helped |
| Apply Self-Align plus SALMON instead of LLaMA-2-Chat-70b alignment | LLaMA-2-70b · TydiQA | GP: 27.9 -> 64.3 | Helped |
| Add PPO training with SALMON | Dromedary-2-70b · TruthfulQA | truthful and informative percentage: 0.89 -> 0.98 | Helped |
| Train with helpful principles rather than the base setup | LLaMA-2-70B with SALMON · HHH Eval Helpful | MC accuracy: 0.90 -> 0.92 | Helped |
| Train with honest principles rather than the base setup | LLaMA-2-70B with SALMON · HHH Eval Honest | MC accuracy: 0.77 -> 0.80 | Helped |
| Train with harmless principles rather than the base setup | LLaMA-2-70B with SALMON · HHH Eval Harmless | MC accuracy: 0.91 -> 0.93 | Helped |
| Post-train with SALMON using helpful principles | LLaMA-2-Chat-70b · MT-Bench | MT: 6.88 -> 6.95 | Helped |
| Post-train with SALMON using helpful principles | LLaMA-2-Chat-70b · MT-Bench | T-2: 6.73 -> 6.72 | Hurt |
| Use honest principles instead of base principles | LLaMA-2-70B with SALMON · HHH Eval Overall | MC accuracy: 0.86 -> 0.88 | Helped |
Surprising findings
- The instructable reward model with a self-praise prohibition principle reverses the adversarial failure (15.0% to 89.8%), showing reward hacking can be countered by textual principles without fresh human preferences.
- LLaMA-2-70b base with SALMON already achieves very high HHH scores, so SFT warm-up was skipped for the HHH experiments.
Limitations
- Dromedary-2 hallucinates unverified information and makes reasoning errors, possibly due to SFT-initialized reward models.
- Crafting robust, non-conflicting principles is difficult and yields unexpected results when principles conflict.
- Randomly sampled principles may be suboptimal; task-dependent principle selection is not solved.
Experimental setup
Datasets: ShareGPT databricks-dolly-15k OpenAssistant Conversations (OASST1) OpenOrca Anthropic HH-RLHF Stanford SHP MATH
Benchmarks: Vicuna-Bench MT-Bench AlpacaEval Big-Bench Hard HumanEval TydiQA TruthfulQA Big-bench HHH Eval Anthropic HH test sets (reward model accuracy)
Metrics: GPT-4 judged score win rate accuracy pass@1 GP (TydiQA) truthful and informative rate MC accuracy
Baselines: LLaMA-2-Chat-70b ChatGPT GPT-4 Text-Davinci-003 Vicuna-33b Guanaco OpenAssistant Humpback LIMA Claude
Builds on: Self-Align Constitutional AI Principle-Driven Self-Alignment PPO QLoRA
Closest papers by meaning
- RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.90)
- Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (ICML 2024 · similarity 0.90)
- Generative RLHF-V: Learning Principles from Multi-modal Human Preference (NeurIPS 2025 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- Preference Ranking Optimization for Human Alignment (AAAI 2024 · similarity 0.90)
- The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.