Research map / Large language models / Alignment and preferences
Position: Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that social choice theory should guide how diverse human feedback is aggregated in RLHF and constitutional AI, proposing RLCHF and simulated collective decision variants.
Problem
RLHF and constitutional AI rely on feedback from diverse, potentially conflicting humans, raising questions about who provides feedback, what format is used, and how it is aggregated into consistent collective choices.
Approach
Frames alignment feedback aggregation as a social choice problem and proposes concrete variants (RLCHF with aggregated rankings or ratings, supervised learning from simulated collective decisions, inference-time simulated collective choice) that insert explicit social welfare or choice functions into the RLHF pipeline, with evaluator features so aggregation is transparent.
Key results
- Illustrates with a toy example (Borda, Instant Runoff, Ranked Pairs disagree) that aggregation rules yield different outcomes on the same rankings.
- Shows via a clone example that Plurality violates independence of clones, with the Indian restaurant winning 48% after a clone is added.
- Argues Borda Count badly violates independence of clones and is implicitly used in some RLHF approaches.
Surprising findings
- Borda Count, implicitly used in some standard RLHF approaches, badly violates independence of clones, which may matter for chatbot responses.
Limitations
- No empirical experiments; proposals are conceptual sketches without implementation or evaluation.
- Many open questions about evaluator representativeness, feature design, and behavioral effects are left unresolved.
Experimental setup
Builds on: RLHF Constitutional AI Direct Preference Optimization Arrow's Impossibility Theorem computational social choice
Closest papers by meaning
- MaxMin-RLHF: Alignment with Diverse Human Preferences (ICML 2024 · similarity 0.88)
- Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? (NeurIPS 2025 · similarity 0.87)
- An Evolutionary Perspective on AI Alignment (Student Abstract) (AAAI 2025 · similarity 0.87)
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning (NeurIPS 2024 · similarity 0.86)
- Moral Alignment for LLM Agents (ICLR 2025 · similarity 0.86)
- Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.