Research map / Large language models / Alignment and preferences
Aligning to Thousands of Preferences via System Message Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes steering LLMs to individual preferences via diverse system messages, releasing the Multifaceted Collection dataset (197k system messages) and training Janus 7B from Mistral 7B v0.2 to generalize to unseen system messages.
Problem
Aligning LLMs to diverse individual preferences without re-training reward models and LLMs for each user; open-source LLMs fail to follow unseen system messages because they are trained with a generic system message.
Approach
Hierarchically synthesize multifaceted user values (dimensions, subdimensions, values) and verbalize combinations as system messages paired with instructions and gold responses, so a single model learns to condition on arbitrary user-specified preferences in the system message at test time.
Key results
- Janus achieves tie+win rates of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4 on Multifaceted Bench human pairwise evaluation.
- Janus 7B reaches a length-controlled win rate of 26.9% on AlpacaEval 2.0, above LLaMA 3 8B Instruct (22.9%) and Mistral 7B Instruct v0.2 (17.1%).
- Janus+DPO 7B averages 4.24 on the Multifaceted Bench LLM-judge score, and Best-of-64 with Janus-RM raises Janus+DPO to 4.31.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train Mistral 7B v0.2 on Multifaceted Collection | Mistral 7B v0.2 · Multifaceted Bench | average score: 3.93 -> 4.17 | Helped |
| Apply DPO to Janus* 7B (ablation) | Janus* 7B · Multifaceted Bench | average score: 4.17 -> 4.24 | Helped |
| Apply ORPO to Janus* 7B instead of the full Janus training setup (ablation) | Janus 7B · Multifaceted Bench | average score: 4.17 -> 4.18 | Mixed |
| Apply Best-of-64 sampling with Janus-RM 7B (ablation) | Janus* 7B · Multifaceted Bench | average score: 4.14 -> 4.28 | Helped |
| Apply Best-of-64 sampling with Janus-RM 7B (ablation) | Janus+DPO 7B · Multifaceted Bench | average score: 4.24 -> 4.31 | Helped |
| Train on multifaceted system messages and multifaceted responses rather than helpful responses without a system message (ablation) | Janus 7B · Multifaceted Bench | average score: 3.88 -> 4.17 | Helped |
| Add multifaceted system messages while retaining multifaceted responses (ablation) | Janus 7B · MT-Bench | score: 7.61 -> 7.74 | Helped |
| Increase training data from 1 to 2 system messages per instruction (ablation) | Janus training · Multifaceted Bench | mf-Koala: 4.36 -> 4.39 | Helped |
| Increase training data from 2 to 3 system messages per instruction (ablation) | Janus training · Multifaceted Bench | mf-FLASK: 4.05 -> 4.39 | Helped |
| Use multifaceted responses instead of helpful responses without a system message (ablation) | Janus 7B · Multifaceted Bench | average score: 3.88 -> 4.01 | Helped |
| Replace multifaceted system messages with a default system message (ablation) | Janus 7B · Multifaceted Bench | average score: 4.17 -> 3.87 | Hurt |
Surprising findings
- Training with diverse system messages also improves general helpfulness: Janus outperforms LLaMA 3 8B Instruct on AlpacaEval 2.0 and Arena Hard Auto despite targeting personalization.
- Janus retains strong performance without any system message at test time, with only a minor drop.
Limitations
- Reliance on synthetic GPT-4-generated training data; full human evaluation of the dataset infeasible.
- Flexible preference specification may increase vulnerability to jailbreaking; safety addressed only partially.
- Capturing implicit preferences from conversation history to form system messages is left out of scope.
Experimental setup
Datasets: Chatbot Arena Conversations Domain-Specific Preference dataset UltraFeedback-binarized-clean Nectar OpenHermesPreferences AlpacaEval 2.0 FLASK Koala MT-Bench Self-Instruct Arena Hard Auto v0.1 RealToxicityPrompts
Benchmarks: Multifaceted Bench AlpacaEval 2.0 MT-Bench Arena Hard Auto v0.1 RealToxicityPrompts Winogender CrowS-Pairs BBQ
Metrics: pairwise human win/tie rate LLM-as-judge score 1-5 length-controlled win rate toxicity fluency perplexity diversity dist-n ROUGE-L Jensen-Shannon distance
Baselines: Mistral 7B Instruct v0.2 LLaMA 3 8B Instruct LLaMA 3 70B Instruct GPT-3.5-Turbo-0125 GPT-4-0613 GPT-4-Turbo-0125 Mixtral 8x7B Instruct v0.1 LLaMA 2 Chat 70B
Builds on: Direct Preference Optimization ORPO Mistral 7B v0.2 Best-of-N sampling
Closest papers by meaning
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.92)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.92)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.92)
- Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment (ICLR 2025 · similarity 0.92)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.91)
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.