Research map / Multi-agent systems and games / LLM agents
Mitigating Social Bias in Large Language Models: A Multi-Objective Approach Within a Multi-Agent Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOMA, a hierarchical multi-agent framework that masks and balances social-group identifiers in prompts (causal-intervention style) to reduce social bias in LLMs while preserving accuracy.
Problem
LLMs produce socially biased outputs, and existing prompt-based ethical debiasing (anti-bias prompting with human values) causes large accuracy degradation (the "alignment tax").
Approach
Instead of prepending ethical instructions to the output-generating model, assistant agents mask bias-related social group identifiers in the input and then add positive counterfactual adjectives for each group, while task agents answer the transformed question without direct exposure to human-value instructions; framed as a causal intervention that breaks the shortcut from group identity to answe
Key results
- MOMA reduces BBQ bias scores by up to 87.7% with at most 6.8% accuracy degradation (abstract).
- On StereoSet intrasentence with GPT-3.5-Turbo, MOMA masking reaches icat 92.63 (+58.1% over baseline 58.60).
- Anti-bias prompting averaged over five prompts reduces Llama-3-8B bias score by 64.6% but accuracy by 37.2% on BBQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use anti-bias prompting instead of standard prompting | Llama-3-8B-Instruct · BBQ | Bias Score: 0.138 -> 0.049 | Helped |
| Use anti-bias prompting instead of standard prompting | Llama-3-8B-Instruct · BBQ | Accuracy: 0.863 -> 0.542 | Hurt |
| Use anti-bias prompting instead of standard prompting | GPT-3.5-Turbo · BBQ | Bias Score: 0.094 -> 0.039 | Helped |
| Use anti-bias prompting instead of standard prompting | GPT-3.5-Turbo · BBQ | Accuracy: 0.840 -> 0.621 | Hurt |
| Apply masking and balancing rather than baseline | MOMA on StereoSet intrasentence · StereoSet intrasentence, GPT-3.5-Turbo | icat: 58.60 -> 92.63 | Helped |
| Apply masking and balancing rather than baseline | MOMA on StereoSet intrasentence · StereoSet intrasentence, Llama-3-8B-Instruct | icat: 66.83 -> 88.23 | Helped |
| Add balancing after masking (ablation) | MOMA · BBQ | Bias Score: about 0.027 | Hurt |
| Add balancing after masking (ablation) | MOMA · BBQ | Accuracy | Hurt |
| Use balancing rather than neutral adjective replacement (ablation) | MOMA balancing · BBQ | Bias Score: 50.2% | Helped |
| Use balancing rather than neutral adjective replacement (ablation) | MOMA balancing · BBQ | Accuracy: 2.9% | Hurt |
| Use unfair positive adjectives rather than neutral replacement (ablation) | MOMA balancing styles · BBQ | Bias Score | Hurt |
| Use fair positive adjectives rather than masking (ablation) | MOMA balancing styles · BBQ | Bias Score | Hurt |
Surprising findings
- Adding positive adjectives without masking (Unfair Positive) worsened bias outcomes, suggesting positive phrases alone can increase group disparities.
- Masking alone is the most effective debiasing variant, while balancing preserves more context at a small bias cost.
Limitations
- Masking loses contextual information and quantifying this loss is left to future work.
- Additional API cost versus single-agent prompting, mainly from few-shot assistant agents.
- Evaluation limited to QA-format datasets (BBQ, StereoSet) and two models; StereoSet intersentence results are inconclusive.
Experimental setup
Datasets: BBQ StereoSet
Benchmarks: BBQ StereoSet
Metrics: bias score accuracy ss lms icat
Baselines: standard prompting chain-of-thought anti-bias prompting society of mind self-consistency
Builds on: chain-of-thought self-consistency society of mind anti-bias prompting
Closest papers by meaning
- Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Models (NeurIPS 2025 · similarity 0.89)
- SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models (AAAI 2024 · similarity 0.89)
- Debiasing Algorithm through Model Adaptation (ICLR 2024 · similarity 0.88)
- Biases Mitigation and Expressiveness Preservation in Language Models: A Comprehensive Pipeline (Student Abstract) (AAAI 2024 · similarity 0.88)
- CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models (ICLR 2025 · similarity 0.88)
- Unbiased Evaluation of Large Language Models from a Causal Perspective (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.