Research map / Large language models / Alignment and preferences
The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PRISM, a dataset linking 1,500 participants' survey profiles and stated preferences to 8,011 rated multi-turn conversations with 21 LLMs, and uses it in three case studies on who gives feedback and how it affects welfare.
Problem
Human feedback for LLM alignment is collected from narrow, weakly documented populations using mostly binary, contextual preference data, so it is unclear whose values are encoded and how sampling choices affect outcomes.
Approach
Pairs detailed sociodemographic and stated-preference survey responses from a geographically diverse, partly census-representative set of participants with their live, fine-grained cardinal ratings of responses from many LLMs across objective, values-guided and controversial conversations, with pseudonymous IDs enabling individual-level attribution.
Key results
- Dataset covers 1,500 participants from 75 countries rating 8,011 conversations with 21 LLMs (9 open-access, 12 commercial API).
- Participant rankings are sensitive to topic and region: e.g., zephyr-7b performs well on controversy prompts but not unguided ones, and gpt models fare significantly worse on PRISM than on ChatbotArena.
- For the US, the welfare-maximising model under meanChoice achieves only 45% choice probability, so no single LLM satisfies most participants.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Controversy-guided priming of prompts | PRISM conversations · PRISM case study I | topic prevalence | Helped |
| Sample size reduced (e.g., 100 vs 100+ people) for choosing a model | model selection by sub-samples · PRISM case study III | probability of choosing worse-welfare LLM | Hurt |
| Sampling exclusively from a specific subgroup (male, white, or 45+) | welfare outcomes for out-group stakeholders · PRISM case study III | out-group welfare | Hurt |
| Adding characters, enumeration, or ending with a question mark in responses | LLM responses scored in PRISM · PRISM case study II regression | score | Helped |
| Adding line breaks to responses | LLM responses scored in PRISM · PRISM case study II regression | score | Hurt |
| Refusals in responses | LLM responses scored in PRISM · PRISM case study II regression | score | Hurt |
Surprising findings
- GPT models rank significantly worse on PRISM than on ChatbotArena, while open models like zephyr-7b rank better.
- Sampling 100 white individuals first-order stochastically dominates a representative sample for overall welfare, yet non-white minorities are worse off under that scheme.
Limitations
- Sample skews White, Western, educated and English-speaking, drawn from a single crowdworking platform (Prolific).
- Cardinal rating scale introduces intrapersonal noise and limited interpersonal comparability; participants may not honestly report preferences.
- Models are pre-aligned and quickly become outdated; only 21 models included, with a limited regression fit (R^2=0.06) for style factors.
Experimental setup
Datasets: ChatbotArena
Benchmarks: PRISM ChatbotArena
Metrics: mean rating mean choice likelihood rank centrality scores first-order stochastic dominance welfare distributions OLS coefficients
Baselines: Chatbot Arena Lmsys-1m WildChat OpenConvos Dices
Builds on: RLHF Chatbot Arena Pairwise Rank Centrality Convergence Voting Dynabench
Closest papers by meaning
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.90)
- Questioning the Survey Responses of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Human Feedback is not Gold Standard (ICLR 2024 · similarity 0.88)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.88)
- Examining Alignment of Large Language Models through Representative Heuristics: the case of political stereotypes (ICLR 2025 · similarity 0.87)
- On Evaluating LLM Alignment by Evaluating LLMs as Judges (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.