Research map / Large language models / AI and society
MAP: Multi-Human-Value Alignment Palette
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAP, which casts multi-value LLM alignment as a KL-regularized constrained problem over user-set value palettes (expected reward thresholds), solved via a primal-dual/dual gradient method, with decoding- and finetuning-based implementations.
Problem
Aligning generative models to multiple, potentially conflicting human values simultaneously, where linear reward weights are hard to choose and trade-offs are opaque.
Approach
Define alignment targets as constraints on expected per-value rewards ("value palette"); the KKT/dual solution is an exponentially tilted distribution p0*exp(lambda^T r), so a single linear-weighted reward realizes the Pareto set, and lambda is found by concave dual gradient ascent; feasibility of a palette is checked by whether the dual converges.
Key results
- On OPT-1.3B conversational task, HHH-50% MAP-D raises Humor from 2.07 to 2.44 and Harmlessness from 0.25 to 0.21 (decoding) while MAP-F reaches 0.66 harmlessness, showing palette-controlled multi-value gains.
- Navigation efficiency of expected reward: MAP-D 55% and MAP-F 52%, versus 21-45% for MORL, DPO and DPO-Soup baselines.
- Five rounds of sequential alignment approach MAP outcomes, while one round is less effective than MAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MAP-D with an HHH-70% palette instead of the original model | OPT-1.3B · Anthropic conversational data | Humor: 2.07 -> 2.49 | Helped |
| Use MAP-D with an HHH-70% palette instead of the original model | OPT-1.3B · Anthropic conversational data | Harmlessness: 0.25 -> 0.66 | Helped |
| Use MAP-D with an HHH-70% palette instead of the original model | OPT-1.3B · Anthropic conversational data | Helpfulness: -1.47 -> -1.29 | Helped |
| Use MAP-D with an HHH-70% palette instead of the original model | Llama2-7B-chat · Anthropic conversational data | Humor: 0.60 -> 2.01 | Helped |
| Use MAP-D with an HHH-80% palette instead of the original model | Llama2-7B-chat · Anthropic conversational data | Harmlessness: 1.25 -> 1.68 | Helped |
| Use MAP-D with a PHH-70% palette instead of the original model | OPT-1.3B · IMDB open generation | Positiveness: 0.52 -> 0.93 | Helped |
| Use MAP-D with a PHH-70% palette instead of the original model | OPT-1.3B · IMDB open generation | Helpfulness: -1.53 -> -1.10 | Helped |
| Use MAP-D with a PHH-70% palette instead of the original model | OPT-1.3B · IMDB open generation | Harmlessness: 0.58 -> 0.74 | Helped |
| Use MAP-D rather than MORL-D | OPT-1.3B · Desirable regime across alignment methods | Navigation Efficiency of expected reward: 21% -> 55% | Helped |
| Use MAP-F rather than MORL-F | OPT-1.3B · Desirable regime across alignment methods | Navigation Efficiency of win rate: 55% -> 70% | Helped |
| Use a single-value palette rather than a multi-value palette | MAP alignment · Anthropic conversational data | Other value rewards | Mixed |
| Run sequential single-value alignments for five rounds instead of one round | OPT-1.3B sequential MAP · Anthropic conversational data | Realized value levels | Helped |
Surprising findings
- The HHH-80% palette was infeasible for OPT-1.3B but feasible for Llama2-7B-chat, indicating model capacity limits achievable value palettes.
- Single-value alignment often degraded other values (e.g., Humor-80% with MAP-F drops Perplexity to -5.85 and Coherence to 0.09).
Limitations
- Experiments are on a small OPT-1.3B and Llama2-7B-chat with reward models that are themselves imperfect proxies (e.g., GPT-2 value heads, humor classifier).
- Llama2-7B-chat experiments only used decoding-stage alignment due to GPU memory limits; no finetuning results for larger model.
- Feasibility of a palette is determined by whether dual iteration converges, and infeasible palettes require user adjustment or heuristic rescaling.
Experimental setup
Datasets: Anthropic harmless data IMDB reviews
Benchmarks: Anthropic harmless prompts (conversational) IMDB prompts (sentiment-controlled open generation)
Metrics: expected reward (realized value level) win rate navigation efficiency
Baselines: MORL-D MORL-F DPO DPO-Soup sequential alignment
Builds on: RLHF DPO Rewarded Soups PPO
Closest papers by meaning
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.90)
- Learning Human-like Representations to Enable Learning Human Values (NeurIPS 2024 · similarity 0.89)
- Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.89)
- Deep Value Benchmark: Measuring Whether Models Generalize Deep values or Shallow Preferences (NeurIPS 2025 · similarity 0.88)
- Position: Challenges and Future Directions of Data-Centric AI Alignment (ICML 2025 · similarity 0.87)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.