Research map / Large language models / AI and society
Position: A Roadmap to Pluralistic Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper proposing a formal roadmap for pluralistic alignment of LLMs: three forms of model pluralism (Overton, steerable, distributional), three pluralistic benchmark classes, and a small empirical check that alignment reduces distributional pluralism.
Problem
Aligning models to serve diverse human values and perspectives is an open problem; current alignment (e.g., RLHF) optimizes for averaged preferences and it is unclear how to define or measure pluralism in AI systems.
Approach
Formalizes pluralism for models as Overton (output the full window of reasonable answers), steerable (faithfully conditioned on attributes), and distributional (match a target population's answer distribution), plus three pluralistic benchmark types (multi-objective, trade-off steerable, jury-pluralistic), and argues standard alignment may reduce distributional pluralism.
Key results
- In the authors' check on GlobalOpinionQA and MPI, almost all pre-aligned models had lower Jensen-Shannon distance to human target distributions than their post-aligned counterparts (GPT-3 on MPI the exception).
- Post-alignment entropy of model answer distributions was reported to drop, consistent with prior work (Santurkar et al., 2023; Durmus et al., 2023).
- The paper provides a hypothesis rather than a conclusive result: current alignment may reduce distributional pluralism for diverse populations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply post-training alignment | LLaMA2-7B · GlobalOpinionQA (Japan) | Jensen-Shannon distance: 0.47 -> 0.57 | Hurt |
| Apply post-training alignment | LLaMA2-7B · GlobalOpinionQA (US) | Jensen-Shannon distance: 0.43 -> 0.56 | Hurt |
| Apply post-training alignment | LLaMA2-7B · GlobalOpinionQA (Germany) | Jensen-Shannon distance: 0.46 -> 0.57 | Hurt |
| Apply post-training alignment | LLaMA2-7B · MPI | Jensen-Shannon distance: 0.37 -> 0.51 | Hurt |
| Apply post-training alignment | LLaMA2-13B · GlobalOpinionQA (Japan) | Jensen-Shannon distance: 0.40 -> 0.55 | Hurt |
| Apply post-training alignment | LLaMA2-13B · GlobalOpinionQA (US) | Jensen-Shannon distance: 0.37 -> 0.53 | Hurt |
| Apply post-training alignment | LLaMA2-13B · GlobalOpinionQA (Germany) | Jensen-Shannon distance: 0.39 -> 0.55 | Hurt |
| Apply post-training alignment | LLaMA2-13B · MPI | Jensen-Shannon distance: 0.42 -> 0.46 | Hurt |
| Apply post-training alignment | Gemma-7B · GlobalOpinionQA (Japan) | Jensen-Shannon distance: 0.33 -> 0.51 | Hurt |
| Apply post-training alignment | Gemma-7B · GlobalOpinionQA (US) | Jensen-Shannon distance: 0.36 -> 0.52 | Hurt |
| Apply post-training alignment | Gemma-7B · GlobalOpinionQA (Germany) | Jensen-Shannon distance: 0.35 -> 0.51 | Hurt |
| Apply post-training alignment | Gemma-7B · MPI | Jensen-Shannon distance: 0.29 -> 0.56 | Hurt |
Surprising findings
- Alignment procedures such as RLHF appear to reduce distributional similarity to human populations across several model families.
Limitations
- Definitions such as the Overton window and target population are hard to operationalize and may require expert or survey effort.
- Empirical evidence is limited to multiple-choice opinion datasets across few model classes and is described as initial.
- Trade-off and welfare-based approaches assume commensurability, which value pluralists may reject.
Experimental setup
Datasets: GlobalOpinionQA Machine Personality Inventory (MPI) OpinionQA
Benchmarks: GlobalOpinionQA MPI
Metrics: Jensen-Shannon distance
Baselines: Pre-aligned vs post-aligned LLMs (Alpaca, Tulu, GPT-3 instruct)
Builds on: RLHF OpinionQA GlobalOpinionQA social welfare functions multi-objective RL
Closest papers by meaning
- Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties (AAAI 2024 · similarity 0.90)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.90)
- Language Model Alignment in Multilingual Trolley Problems (ICLR 2025 · similarity 0.89)
- MAP: Multi-Human-Value Alignment Palette (ICLR 2025 · similarity 0.89)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.88)
- Pairwise Calibrated Rewards for Pluralistic Alignment (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.