Research map / Large language models / Alignment and preferences
Panacea: Pareto Alignment via Preference Adaptation for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Panacea trains a single LLM whose behavior is steered at inference by a preference vector injected as singular values of an SVD-LoRA, learning the whole Pareto front for multi-dimensional alignment.
Problem
Scalar preference labels oversimplify multi-dimensional, heterogeneous human preferences, causing label conflicts and a single model that cannot cover diverse trade-offs (e.g., helpfulness vs. harmlessness).
Approach
Embed a low-dimensional preference vector as scaled singular values in every SVD-LoRA layer (with learnable per-matrix scaling), train end-to-end on randomly sampled preference vectors with linear scalarization or Tchebycheff aggregation, so one model recovers the Pareto front and adapts online.
Key results
- On HH with Llama1-ft RLHF, Panacea hypervolume 915.04 vs RS 517.28 (LS).
- On Chat 10-dim with Llama3-Instruct SFT, Panacea hypervolume 0.12 vs RS 0.01, described as an order-of-magnitude improvement.
- Panacea learns a better front than discrete policy solutions (11 separate models) on HH while using a single model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Rewarded Soups with Panacea (LS) | HH, Llama1-ft, RLHF · HH | hypervolume: 517.28 -> 915.04 | Helped |
| Replace Rewarded Soups with Panacea (LS) | HH, Llama2-ft, DPO · HH | hypervolume: 0.318 -> 0.337 | Helped |
| Replace Rewarded Soups with Panacea (LS) | HHC, Llama2-ft, RLHF · HHC | hypervolume: 13519 -> 17097 | Helped |
| Replace Rewarded Soups with Panacea (LS) | Chat 3-dim, SFT · Chat 3-dim | hypervolume: 0.29 -> 0.50 | Helped |
| Replace Rewarded Soups with Panacea (LS) | Chat 4-dim, SFT · Chat 4-dim | hypervolume: 0.14 -> 0.38 | Helped |
| Replace Rewarded Soups with Panacea (LS) | Chat 5-dim, SFT · Chat 5-dim | hypervolume: 0.08 -> 0.33 | Helped |
| Replace Rewarded Soups with Panacea (LS) | Chat 10-dim, SFT · Chat 10-dim | hypervolume: 0.01 -> 0.12 | Helped |
| Use a fixed preference-vector scaling factor instead of a learnable per-weight-matrix factor (ablation) | Panacea · HH | Pareto-front performance | Hurt |
| Adapt only the MLP layers (ablation) | Panacea · HH | Pareto-front performance | No clear effect |
| Adapt only the self-attention layers (ablation) | Panacea · HH | Pareto-front performance | No clear effect |
| Adapt only the first 10 layers (ablation) | Panacea · HH | Pareto-front performance | No clear effect |
| Adapt only the last 10 layers (ablation) | Panacea · HH | Pareto-front performance | Hurt |
Surprising findings
- Panacea learns a better and smoother Pareto front than the DPS upper-bound baseline that trains a separate model per preference vector, attributed to positive transfer across dimensions.
- Fixed (non-learnable) preference scaling factors ranging from 1 to 1e-5 all produce significantly worse fronts than learned scaling.
Limitations
- Tchebycheff aggregation is less suitable for RLHF because the ideal maximum reward is hard to determine.
- Reward models for HH are self-trained and their scale is adjusted by a factor of 5 to match open-source models, and HH/HHC use fine-tuned base models rather than released chat models.
Experimental setup
Datasets: BeaverTails Alpaca
Benchmarks: BeaverTails (HH, HHC) Alpaca-based chat multi-dimensional test sets
Metrics: hypervolume inner product sparsity spacing DPO implicit reward accuracy SFT loss
Baselines: Rewarded Soups Discrete Policy Solutions Llama3-Instruct SFT per-dimension models
Builds on: LoRA AdaLoRA Rewarded Soups Safe-RLHF DPO
Closest papers by meaning
- Leveraging robust optimization for llm alignment under distribution shifts (NeurIPS 2025 · similarity 0.93)
- Sequential Preference Optimization: Multi-Dimensional Preference Alignment with Implicit Reward Modeling (AAAI 2025 · similarity 0.92)
- TODO: Enhancing LLM Alignment with Ternary Preferences (ICLR 2025 · similarity 0.92)
- Robust LLM Alignment via Distributionally Robust Direct Preference Optimization (NeurIPS 2025 · similarity 0.92)
- ComPO: Preference Alignment via Comparison Oracles (NeurIPS 2025 · similarity 0.92)
- Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.