Research map / Large language models / Alignment and preferences
Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior
Dalia Ali, Dora Zhao, Allison Koenecke, Orestis Papakyriakopoulos
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 5-point Likert ratings instead of binary (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.198 -> -0.242 | Improved |
| use 5-point Likert ratings instead of 3-point (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.225 -> -0.242 | Improved |
| preserve all ratings instead of majority vote (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.158 -> -0.242 | Improved |
| preserve all ratings instead of full consensus (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.039 -> -0.242 | Improved |
| preserve all ratings instead of random selection (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.146 -> -0.242 | Improved |
| preserve all ratings instead of averaging (ablation) | DPO toxicity fine-tuning · Toxicity | toxicity effect size: -0.229 -> -0.242 | No clear effect |
| use DPO instead of GRPO for multi-objective alignment | Toxic+EA training · Toxicity | toxicity effect size: -0.020 -> -0.159 | Improved |
| use DPO instead of GRPO for multi-objective alignment | Toxic+EA training · Emotional awareness | EA effect size: 0.029 -> 0.084 | Improved |
| single-objective toxicity DPO instead of multi-objective Toxic+EA (ablation) | DPO · Toxicity | toxicity effect size: -0.159 -> -0.243 | Improved |
| single-objective EA DPO instead of multi-objective Toxic+EA (ablation) | DPO · Emotional awareness | EA effect size: 0.084 -> 0.083 | No clear effect |
| fine-tune on Liberal instead of Conservative feedback | DPO · Emotional awareness | pooled effect: 0.049 | Improved |
| fine-tune on White instead of Black feedback | DPO · Emotional awareness | pooled effect: 0.046 | Improved |
Closest papers by meaning
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.91)
- Position: A Roadmap to Pluralistic Alignment (ICML 2024 · similarity 0.90)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.90)
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.90)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.90)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
