Research map / Large language models / Instruction and fine-tuning
Biases Mitigation and Expressiveness Preservation in Language Models: A Comprehensive Pipeline (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage pipeline that debiases BERT-large via continuous prefix-tuning (frozen PLM) with a Wasserstein-based internal debiasing loss and KL alignment, then applies causal intervention with counterfactual augmentation during downstream fine-tuning.
Problem
PLMs encode social (gender) biases, and existing debiasing methods either fully fine-tune the PLM (costly, harming expressiveness) or ignore biases reintroduced during downstream task fine-tuning.
Approach
Combine prefix-tuning with the PLM frozen for internal debiasing (Wasserstein distance between attribute and neutral word embeddings plus a KL/hidden-state alignment term to preserve expressiveness) with a causal-intervention invariance objective on counterfactually augmented downstream data.
Key results
- Ours achieves CrowS-Pairs 53.32 vs 60.34 for original BERT and 43.57 for Context-Debias, with LMS 90.02 vs 90.44 original.
- Downstream accuracy improves over original BERT on SST-2 (0.933 vs 0.924) and RTE (0.560 vs 0.527), and CoLA Mcc 0.633 vs 0.588.
- Context-Debias yields a CoLA Mcc of 0, indicating collapse on that task, while the proposed pipeline preserves it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · SST-2 | accuracy: 0.927 -> 0.933 | Helped |
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · RTE | accuracy: 0.487 -> 0.560 | Helped |
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · CoLA | MCC: 0 -> 0.633 | Helped |
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · Language modeling | LMS: 84.420 -> 90.019 | Helped |
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · Gender bias | SS: 64.300 -> 61.028 | Helped |
| apply the two-stage debiasing pipeline | BERT-LARGE-UNCASED · CrowS-Pairs | score: 60.34 -> 53.32 | Helped |
Surprising findings
- Context-Debias obtains the lowest CrowS-Pairs score but its CoLA Mcc collapses to 0 and LMS drops, suggesting debiasing can damage expressiveness.
Limitations
- Evaluation limited to gender bias and binary attribute setting
- Only three GLUE tasks and a single base model (BERT-large-uncased) evaluated
- Student abstract with limited experimental detail; compared mainly against one baseline (Context-Debias)
Experimental setup
Datasets: News-Commentary v15 SST-2 RTE CoLA SEAT StereoSet CrowS-Pairs
Benchmarks: SEAT CrowS-Pairs GLUE SST-2 GLUE RTE GLUE CoLA
Metrics: SEAT effect size (C6, C7, C8) Stereotype Score (SS) CrowS-Pairs score Language Modeling Score (LMS) accuracy Matthews correlation
Baselines: Context-Debias Original BERT-large-uncased
Builds on: prefix-tuning Context-Debias
Closest papers by meaning
- Debiasing Algorithm through Model Adaptation (ICLR 2024 · similarity 0.91)
- Collapsed Language Models Promote Fairness (ICLR 2025 · similarity 0.90)
- The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models (ICLR 2024 · similarity 0.90)
- Self-correcting for Debiasing Large Language Models (ICML 2026 · similarity 0.90)
- LIDAO: Towards Limited Interventions for Debiasing (Large) Language Models (ICML 2024 · similarity 0.89)
- Prompting Fairness: Integrating Causality to Debias Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.