Research map / Large language models / Language, safety and interpretability

More of the Same: Persistent Representational Harms Under Increased Representation

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GAS(P), a text-based evaluation methodology that surfaces distribution-level gender representational differences in LLM-generated personas and biographies when gender is unprompted, applied to GPT-3.5, GPT-4o-mini, and Llama-3.1-70B across 63 occupations.

Problem

Existing bias evaluations require explicit group specification in prompts, leaving unexamined how groups are represented (not just who is represented) in LLM text generated without group specification, such as gendered occupational portrayals.

Approach

Combine a pronoun/honorific-based Gender Association Method, a calibrated Marked Words method (hybrid English + topic prior with regularization), and a Subset Representational Bias Score based on Chamfer distance over word embeddings to test whether unprompted gender-associated word distributions resemble explicitly prompted gender distributions.

Key results

• Models generate more women than BLS data for most occupations; female-dominated occupations reach 90-100% women in many cases, and GPT-4o-mini and Llama-3.1 show higher female share than GPT-3.5.
• SRB Scores differ significantly between associated women and men across all occupations and models (p < 0.05), indicating unprompted gendered descriptions mirror prompted gender differences.
• Clustering of 1500 clusters found 30 clusters at >=50% prevalence difference; most favor women, including clusters for 'yoga', 'empathy', 'awards', 'inspire', 'diversity', and 'advocate', and one cluster for 'woman' and 'female' with no male counterpart.

Measured effects reported in the paper

Surprising findings

• Non-binary representation is 0% for all occupations in GPT-3.5 and Llama-3.1, showing persistent under-representation.
• The SRB similarity to specified gender increased from GPT-3.5 to GPT-4o-mini, suggesting that newer models amplify rather than remove gendered representational differences.

Limitations

• Gender distributions are from U.S. BLS and may not reflect global occupational distributions.
• Generating enough generations per gender is computationally expensive; the study required about 1000 generations per occupation and prompt.
• The Gender Association Method does not capture all generations (80-98%), and some generations may be misclassified; clustering analysis is restricted to predefined criteria.

Experimental setup

Datasets: Winogender occupations BLS 2024 occupational gender statistics Brown corpus (NLTK)

Benchmarks: BLS 2024 occupational gender distribution

Metrics: percentage of women per occupation SRB Score t-test p-values Gender Association accuracy Calibrated Marked Words z-scores

Baselines: Marked Personas GenBiT WEAT

Builds on: Marked Personas Fightin' Words Word2Vec Winogender

Closest papers by meaning

• Diverse Yet Biased: Towards Mitigating Biases in Generative AI (Student Abstract) (AAAI 2024 · similarity 0.88)
• Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.87)
• SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.87)
• Examining Alignment of Large Language Models through Representative Heuristics: the case of political stereotypes (ICLR 2025 · similarity 0.86)
• StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs (NeurIPS 2026 · similarity 0.86)
• Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency (NeurIPS 2024 · similarity 0.86)