Research map / Learning theory and algorithms / Fairness
First-Person Fairness in Chatbots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a counterfactual, LLM-as-research-assistant (LMRA) method to measure first-person name-based demographic bias in ChatGPT across 66 real-world tasks, finding small harmful gender stereotype rates and reduced bias after post-training RL.
Problem
Chatbot fairness toward the users themselves (first-person fairness) based on demographic cues in user names is under-studied, since chatbot tasks are open-ended and differ from classic institutional decision-making fairness settings.
Approach
Replay real chat prompts with varied user names (and Custom Instructions/Memory profiles) and use an LLM research assistant (LMRA) to rate harmful stereotype pairs with forward/reverse net scoring, enumerate statistically significant "axes of difference," and compare pre- vs post-RL models.
Key results
- Harmful gender stereotype rates are below 0.1% on random chats for all models, and above 2% on the 'write a story' task for GPT-3.5-turbo; GPT-4o-mini shows a mean of under 1% across tasks.
- LMRA gender ratings correlate with mean human ratings at Pearson r=0.86 (90.3% sign alignment), while race (Asian, Black, Hispanic) and feature correlations are weaker (r from 0.34 to 0.75).
- Post-training with RL reduces harmful gender stereotype ratings; the slope of post-RL vs pre-RL ratings is 0.21 (95% CI 0.17-0.24), i.e., roughly 3-12x smaller bias after post-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply post-training after SFT, including RL | GPT-3.5/4 models · 19 selected gender-bias tasks | harmful gender stereotype rating: The slope of the best-fit line is $0.21$ (95\% CI: 0.17, 0.24); all tasks were below the 45-degree $y=x$ line | Helped |
| use Memory rather than Custom Instructions to encode names | GPT-4o-mini · 66 tasks | harmful stereotype rating: 2.15 (95% CI: $1.98$, $2.32$) | Hurt |
| use direct female profile instead of female-male name-based profiles | ChatGPT model · 12 tasks | harmful stereotype rating: correlation 0.98; direct-profile harm ratings are lower | Helped |
| use GPT-4o-mini as the LMRA instead of GPT-4o | LMRA human-rating evaluation · human rating correlation | correlation with human ratings | Hurt |
| hide group identities and require matching responses to groups | LMRA harmful-stereotype evaluation · harmful stereotype assessment | overestimation of harmful stereotypes | Helped |
Surprising findings
- LMRA ratings were overly sensitive to inconsequential differences, e.g., rating one response as using simpler language with 100% probability where humans preferred the other only 55%.
- Direct gender profiles produced lower harm ratings than name-based profiles, despite a 0.98 correlation across tasks.
Limitations
- Name counterfactuals do not capture writing style or topic differences between user groups.
- Only English prompts, binary gender and four races, and U.S.-centric name statistics are studied.
- LMRA is overly sensitive on feature labels and only weakly correlated with human judgments for race and features.
Experimental setup
Datasets: LMSYS WildChat private ChatGPT user prompts (1.1M) Social Security Administration baby names name lists from prior gender-bias LM study
Benchmarks: LMSYS prompts WildChat prompts private ChatGPT chats
Metrics: harmful stereotype rate (net forward minus reverse) Pearson correlation with human ratings sign alignment with human ratings response quality differences axes of difference frequency
Baselines: GPT-3.5-turbo GPT-4 turbo GPT-4o o1-preview o1-mini
Builds on: counterfactual fairness evaluation FairPair LLM-based feature description of text differences
Closest papers by meaning
- FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs (ICLR 2025 · similarity 0.90)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.88)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.86)
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.85)
- Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values (NeurIPS 2025 · similarity 0.85)
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.