Research map / Large language models / Language, safety and interpretability

Probing Social Bias in Labor Market Text Generation by ChatGPT: A Masked Language Model Approach

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRISM, a masked-language-model ranking bias score using validated social-science word lists, and uses it to show ChatGPT (GPT-3.5 Turbo) job applications reproduce and amplify gendered language from 33K LinkedIn job postings.

Problem

Measuring gender bias in LLM-generated text in high-stakes labor market settings, where existing bias measures rely on static word counts or embeddings that ignore sentence context and need labeled data.

Approach

PRISM masks each word in turn, uses an MLM's predicted probabilities for feminine vs masculine word lists, converts them to within-text ranks, and takes the rank-mean difference as a context-sensitive, training-free bias score; the paper also derives a rank-test null distribution.

Key results

• PRISM achieves AUC 0.97 classifying gender on the BIOS dataset, above the unigram and word-embedding baselines from Dhamala et al.
• Spearman correlation of 0.853 between PRISM scores and six expert labelers' bias ratings on job advertisements.
• Job posting and ChatGPT application bias scores correlate positively across dimensions, with Social Characteristics r=0.777 and Role Description r=0.708.

Measured effects reported in the paper

Surprising findings

• Role Description bias shifts toward the feminine direction in ChatGPT applications, despite the postings being strongly masculine-skewed.
• Bias magnitude decreases for Psychological Cues and Role Description, while variance drops in all dimensions.

Limitations

• Tokenization splits words into subwords, which affects word-list matching (e.g., 'limitless').
• Results are English-only and depend on the inherent biases of the public pretrained MLMs used for scoring.
• Job application generation uses only basic prompts; effects of varied prompts are untested.

Experimental setup

Datasets: LinkedIn job postings (33K+, Kaggle) BIOS dataset

Benchmarks: BIOS dataset gender classification Human expert labels on job advertisements

Metrics: AUC Spearman correlation Pearson correlation Wilcoxon signed-rank test Levene's test

Baselines: Dhamala et al. unigram matching Gender-Wavg word embedding score Gender-Max word embedding score

Builds on: Dhamala et al. 2021 bias measurement Gaucher et al. 2011 gendered wording Bem 1974 sex role inventory

Closest papers by meaning

• SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.88)
• Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.88)
• Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency (NeurIPS 2024 · similarity 0.86)
• More of the Same: Persistent Representational Harms Under Increased Representation (NeurIPS 2025 · similarity 0.86)
• Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.85)
• Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Models (NeurIPS 2025 · similarity 0.85)