Research map / Large language models / Language, safety and interpretability
The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Integrated Gap Gradients (IG^2) to locate social bias neurons in masked language models by attributing the logits gap between demographic pairs, and a training-free Bias Neuron Suppression (BNS) method that zeroes those neurons to reduce bias.
Problem
Social biases in pre-trained language models are detected only via black-box probing and mitigated by costly fine-tuning or pre-training on anti-stereotypical data; the internal units responsible for biased demographic output gaps are unknown.
Approach
Adapt integrated gradients to attribute the logits gap between two demographics (rather than a single output) to FFN neurons, then suppress the top-attributed neurons at inference time without any retraining; also analyze how bias neurons shift layers after debiasing.
Key results
- BNS on RoBERTa-base reaches ICAT 84.79 while Union_IG gets stereotype score 53.82 but LMS falls from 91.70 to 30.61.
- Suppressing IG^2-identified neurons decreases the logits gap by 22.98% for Gender-Negative on FairBERTa, while amplifying them increases it by 29.05%; random neurons have minimal effect.
- Bias neurons in RoBERTa are concentrated in the deepest layers; after FairBERTa debiasing, the count is nearly unchanged (10.35 to 10.06 on average) but about one third shift to the shallowest three layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Suppress IG^2-identified social bias neurons | FairBERTa · Gender-N | logits gap: decreases 22.98% | Helped |
| Amplify IG^2-identified social bias neurons | FairBERTa · Gender-N | logits gap: increases 29.05% | Hurt |
| Suppress the union of neurons identified by vanilla IG for each demographic | RoBERTa-Base with Union_IG · StereoSet intrasentence | language modeling score (LMS): 91.70 -> 30.61 | Hurt |
| Suppress the union of neurons identified by vanilla IG for each demographic | RoBERTa-Base with Union_IG · StereoSet intrasentence | stereotype score (SS): 53.82 | Helped |
| Use BNS to suppress identified social bias neurons | BNS · StereoSet intrasentence | idealized CAT score (ICAT): 84.79 | Helped |
| Manipulate randomly selected neurons instead of IG^2-identified neurons | FairBERTa · selected Unfair Targets | logits gap | No clear effect |
| Pre-train RoBERTa on anti-stereotypical data | RoBERTa · social bias neuron count | average number of social bias neurons: from 10.35 to 10.06 | No clear effect |
| Pre-train RoBERTa on anti-stereotypical data | RoBERTa · social bias neuron distribution across layers | percentage of neurons in the two shallowest layers: from 2.9% (RoBERTa) to 26.0% (FairBERTa) | Mixed |
Surprising findings
- Debiasing by retraining on anti-stereotypical data mainly relocates bias neurons to shallow layers rather than reducing their number.
Limitations
- BNS zeroes activations of selected neurons, a coarse suppression that may be improved by finer methods.
- Evaluation relies on StereoSet intrasentence subset, limited to masked language models.
- Verification uses a sampled set of judged unfair targets and templates.
Experimental setup
Datasets: StereoSet ParaRel-inspired templates (constructed)
Benchmarks: StereoSet intrasentence
Metrics: LMS SS ICAT logits gap change ratio
Baselines: FairBERTa DPCE AutoDebias Union_IG
Builds on: Integrated Gradients ParaRel StereoSet
Closest papers by meaning
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.90)
- Biases Mitigation and Expressiveness Preservation in Language Models: A Comprehensive Pipeline (Student Abstract) (AAAI 2024 · similarity 0.90)
- Interpretable Debiasing of Vision-Language Models for Social Fairness (CVPR 2026 · similarity 0.90)
- Robust Evaluation Measures for Evaluating Social Biases in Masked Language Models (AAAI 2024 · similarity 0.88)
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.88)
- Mitigating Social Bias in Large Language Models: A Multi-Objective Approach Within a Multi-Agent Framework (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.