Research map / Large language models / AI and society
BeyondGender: A Multifaceted Bilingual Dataset for Practical Sexism Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BeyondGender, a bilingual (English/Chinese) sexism dataset with multifaceted labels (sexism, target gender, phrasing, misogyny, misandry), and benchmarks MLMs and LLMs on it.
Problem
Existing sexism datasets overlook misandry and misogyny directed at men, and their overly broad annotations label gender-equality discussions and misconduct-driven hostility as sexism, causing false positives.
Approach
A carefully defined multi-level annotation codebook (sexism first, then gender, phrasing, misogyny, misandry) applied to bilingual English and Chinese data from YouTube plus EDOS and SWSR, explicitly including misandry and excluding non-gender hatred and debate-related content.
Key results
- English misogyny detection reaches F1 0.87 with the best MLMs (RoBERTa/DeBERTa-level results), while misandry detection is weak in English (F1 around 0.2-0.3).
- Chinese misandry detection reaches F1 0.62 with BERT, and LLMs significantly outperform MLMs on Chinese sexism detection (ChatGLM 0.82 vs BERT 0.56).
- Models frequently mislabel hostile but non-gender-hateful comments as misogyny/misandry and miss mild or benevolent sexism, with at least 1/7 of Chinese mild sexism undetected.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an LLM instead of an MLM for sexism detection | Chinese test set · Chinese test set | F1: .56 -> .82 | Helped |
| Use an MLM instead of an LLM for target-gender classification | Chinese test set · Chinese test set | F1: .53 -> .76 | Helped |
| Use an MLM instead of an LLM for phrasing classification | Chinese test set · Chinese test set | F1: .64 -> .75 | Helped |
| Use an MLM instead of an LLM for misogyny detection | Chinese test set · Chinese test set | F1: .56 -> .61 | Helped |
| Use an MLM instead of an LLM for misandry detection | Chinese test set · Chinese test set | F1: .55 -> .62 | Helped |
Surprising findings
- Alpaca reaches accuracy comparable to other LLMs on English sexism detection but predicts every instance as sexist-labeled for phrasing and misogyny, so accuracy is misleading.
- Misogyny in Chinese data is often expressed by women (e.g., self-loathing), while misandry is more prominent than misogyny in Chinese data.
Limitations
- Misandry samples are scarce in English, which limits the reliability of detection models for that label.
- Models are trained separately per label, so predictions are not consistent across the label set.
- Data is drawn mainly from YouTube and prior datasets from recent decades, not representing historical or future trends.
Experimental setup
Datasets: EDOS SWSR YouTube comments
Benchmarks: BeyondGender English test set BeyondGender Chinese test set
Metrics: Precision Recall F1-score Accuracy
Baselines: BERT RoBERTa DeBERTa ChatGPT ChatGLM Baichuan Llama Alpaca
Builds on: Ambivalent Sexism theory EDOS SWSR
Closest papers by meaning
- Are Models Biased on Text without Gender-related Language? (ICLR 2024 · similarity 0.85)
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.84)
- StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs (NeurIPS 2026 · similarity 0.83)
- More of the Same: Persistent Representational Harms Under Increased Representation (NeurIPS 2025 · similarity 0.83)
- Debiasing Algorithm through Model Adaptation (ICLR 2024 · similarity 0.83)
- A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.