Research map / Large language models / Language, safety and interpretability
Are Models Biased on Text without Gender-related Language?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UnStereoEval, a PMI-based framework and ChatGPT-generated benchmarks that test whether 28 LMs show gender preferences on stereotype-free sentences, finding fairness of only 9%-41%.
Problem
Gender bias in language models is usually measured on stereotypical sentences whose bias is attributed to gendered training correlations; it is unclear whether models remain fair in non-stereotypical, gender-neutral settings.
Approach
Filter sentences so that every word has minimal PMI-based gender co-occurrence in the pretraining corpus (The Pile), generate gender-invariant sentence pairs with ChatGPT around seed words, and measure the fraction of pairs where the LM's log-likelihood gap is within a threshold (Unstereo Score).
Key results
- All 28 tested models show low fairness, with US scores between 9% and 41% on the stereotype-free benchmarks.
- Models consistently prefer male completions on filtered Winobias and Winogender, with preference disparity beyond 40% for all models.
- Tightening the PMI constraint (eta from infinity to 0.65) changes fairness by only small amounts (e.g., |Delta| up to 6.23 on Winobias), so gender co-occurrence does not explain the bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate language models on stereotype-free sentence pairs | 28 tested language models · USE 5, Winobias, Winogender, and other benchmarks | Unstereo Score: 9%–41% fairness | Hurt |
| Reduce the maximum allowed gender-correlation threshold to η=0.65 | language models evaluated on USE 5 · USE 5 | Unstereo Score fairness gap: |Δ0.65| ≤ 1.33% | No clear effect |
| Increase model size | six language-model families · gender-bias benchmarks | Unstereo Score | No clear effect |
| Deduplicate pretraining data (ablation) | Pyth models · USE 5, USE 10, USE 20, Winobias, and Winogender | Unstereo Score: 6.73; -17.14; 9.35 | Mixed |
| Evaluate in filtered Winobias and Winogender | all evaluated models · Winobias and Winogender | gender preference disparity: greater than 40% | Hurt |
Surprising findings
- Fairness does not improve when filtering out gender co-occurring words, suggesting bias is not solely due to gendered lexical cues.
- Pile-trained models (GPT-J, Pythia) and MPT/OLMo tend to favor female completions on USE, while OPT favors male completions.
Limitations
- Focuses only on binary gender with English he/she pronouns.
- Benchmarks generated with a single model (ChatGPT), which may introduce artifacts.
- Relies on The Pile as a proxy for pretraining data of models not trained on it.
Experimental setup
Datasets: The Pile Winobias Winogender
Benchmarks: USE-5 USE-10 USE-20 Winobias Winogender
Metrics: Unstereo Score fairness gap preference disparity AuFC
Baselines: Winobias Winogender BUG
Builds on: Winobias Winogender The Pile
Closest papers by meaning
- Debiasing Algorithm through Model Adaptation (ICLR 2024 · similarity 0.89)
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.88)
- Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency (NeurIPS 2024 · similarity 0.88)
- StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs (NeurIPS 2026 · similarity 0.88)
- More of the Same: Persistent Representational Harms Under Increased Representation (NeurIPS 2025 · similarity 0.87)
- Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.