Research map / Large language models / Language, safety and interpretability

StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces StereoTales, a 10-language open-ended story-generation dataset (~650k stories from 23 LLMs) with statistical discovery of demographic associations, harmfulness rated by 247 humans and by the same LLMs.

Problem

Existing social bias benchmarks are English-centric, template-based, or test recognition of pre-specified stereotypes, leaving bias emergence in open-ended multilingual LLM generation largely uncharacterized.

Approach

Discover over-represented demographic associations from free-form stories (contingency tables with Fisher tests, Cramér's V and lift) rather than predefined stereotypes, then judge harmfulness with both a human panel and cross-model LLM self-evaluation, across 10 manually translated languages.

Key results

• Every one of 23 evaluated models emits harmful associations, ranging from 20 (MiniMax M2.5) to 45 (Gemini 3 Flash) per model; 1,580 unique associations overall.
• LLM and human harmfulness ratings are broadly aligned (Spearman rho=0.62, Pearson r=0.64), with LLMs ~0.11 Likert points more conservative.
• Models underestimate harm on age, education, income, religion and immigration status stereotypes while overrating gender-related ones (+0.12 on gender).

Surprising findings

• Scale and capability show no mitigation of harmful generation; weak positive correlation between ELO and harmful association count.
• Harmful associations are more language-local than benign ones, with the prompt language shifting which groups are targeted.

Limitations

• Human panel is single-country (UK), English-language, and 89.9% White, so harm ratings may miss locally salient harms; counts are conservative lower bounds.
• Pairwise co-occurrence analysis cannot decompose attribute mediation or causal chains.
• Language coverage excludes sub-Saharan African, Southeast Asian and Indigenous languages.

Experimental setup

Datasets: StereoTales prompts (19 attributes, 79 values, 36 scenarios)

Benchmarks: StereoTales

Metrics: lift bias-corrected Cramér's V Spearman correlation Pearson correlation median Likert harmfulness

Baselines: BBQ BOLD Marked Personas SeeGULL SHADES Phare

Builds on: Phare BADF BBQ BOLD

Closest papers by meaning

• Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency (NeurIPS 2024 · similarity 0.91)
• Investigating and Mitigating Undesirable Biases in Large Language Models (AAAI 2025 · similarity 0.89)
• SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.89)
• Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.88)
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs (ICLR 2024 · similarity 0.88)
• Robust Evaluation Measures for Evaluating Social Biases in Masked Language Models (AAAI 2024 · similarity 0.87)