Research map / Large language models / Alignment and preferences
Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework
Nora Petrova, Andrew Gordon, Enzo Blindow
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Overall Winner dimension instead of Trust, Ethics & Safety | pairwise human preference evaluation · HUMAINE | tie rate (discriminative power): 65% -> 10% | Improved |
| compare older (55+) vs younger (18-34) raters | model rankings · HUMAINE | tie rate: 9.7% -> 12.5% | Hurt |
| vary rater age group | model rankings · HUMAINE | average rank shift: age 2.8 vs ethnicity 1.3 vs politics 1.5 | Mixed |
| break results down by dimension rather than overall | gemini-2.5-pro ranking · HUMAINE | rank: top rank across all dimensions | No clear effect |
| break results down by dimension | grok-3 / magistral-medium ranking · HUMAINE | rank across dimensions: grok-3 2nd on Core Task vs 8th on Communication Style; magistral 2nd on Interaction Fluidity vs 12th on Trust | Mixed |
| move age cohort from 18-34 to 55+ | magistral-medium-2506 · HUMAINE | rank (US): 1st -> 5th | Hurt |
| add interaction terms to additive demographic model (ablation) | hierarchical BTD · HUMAINE | variance explained by interaction: 7-41% US, 29-50% UK | Mixed |
Closest papers by meaning
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.92)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.91)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.89)
- InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback (NeurIPS 2025 · similarity 0.89)
- Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes (ICML 2025 · similarity 0.89)
- ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
