Research map / Large language models / Language, safety and interpretability
FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FairMT-Bench, a multi-turn fairness benchmark (FairMT-10K built from RedditBias, SBIC, HateXplain across six tasks, plus a harder FairMT-1K subset) that evaluates LLM bias accumulation across dialogue turns using GPT-4 and Llama-Guard-3 judges.
Problem
Existing fairness benchmarks for LLMs focus on single-turn dialogues, leaving bias accumulation and contextual interference in multi-turn conversations under-evaluated.
Approach
A task taxonomy across three stages (context understanding, interaction fairness, fairness trade-off) with six multi-turn tasks, built from existing bias datasets, to measure how bias ratios change across turns and versus single-turn prompts.
Key results
- On FairMT-10K, bias ratios generally rise with dialogue turns, and most models show higher bias in multi-turn than single-turn settings (e.g., Llama-2 and Llama-3.1 show large increases).
- GPT-4 judge agreement with human annotators reaches 95.69% accuracy and 93.97% F1 overall.
- On the harder FairMT-1K, all 15 evaluated models show substantial bias ratios, e.g., Llama-3.1-8B-Instruct averages 80.00% and Gemma-2-27b-it averages 25.35%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-turn dialogue rather than the final prompt as a single-turn input | LLMs evaluated on FairMT-Bench · FairMT-10K, stereotype and toxicity | bias ratio: higher bias ratio in multi-turn dialogues than in single-turn ones across Stereotype and Toxicity for all LLMs except Ge | Hurt |
| Increase dialogue turn count | LLMs evaluated on FairMT-10K · FairMT-10K, stereotype and toxicity | bias ratio | Hurt |
| Use multi-turn dialogue history in Fixed Format rather than the corresponding single-turn input | Gemma-7B-it · FairMT-10K, Fixed Format | bias ratio: 83% and 77% reduction in bias | Helped |
| Increase model size | models evaluated on FairMT-1K · FairMT-1K, Scattered Questions, Fixed Format, and Negative Feedback | bias ratio | Helped |
| Use Anaphora Ellipsis or Interference from Misinformation instead of less challenging tasks | LLMs evaluated on FairMT-10K · FairMT-10K | bias ratio | Hurt |
Surprising findings
- Gemma-7b-it shows lower bias in multi-turn than single-turn dialogues, driven by a large drop on the Fixed Format task.
- Larger models do not uniformly reduce bias; some tasks show bias decreasing with size while others (e.g., Qwen2.5 small models) show high bias ratios.
Limitations
- GPT-4 judge tends toward over-protection, flagging some responses that human annotators label as unbiased.
- Llama-Guard-3 is less sensitive to implicit biases than GPT-4.
- Human validation is limited to a sample judged by three annotators and only covers six tasks.
Experimental setup
Datasets: RedditBias SBIC HateXplain FairMT-10K
Benchmarks: FairMT-10K FairMT-1K MT-Bench-101
Metrics: bias ratio accuracy precision recall F1
Baselines: ChatGPT Llama-2-7b-chat Llama-2-13b-chat Llama-3.1-8B-Instruct Mistral-7B-Instruct-v0.3 Gemma-7b-it
Builds on: Chain of Attack RedditBias SBIC HateXplain MT-Bench-101
Closest papers by meaning
- First-Person Fairness in Chatbots (ICLR 2025 · similarity 0.90)
- A Statistical Framework for Ranking LLM-based Chatbots (ICLR 2025 · similarity 0.89)
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (ICML 2024 · similarity 0.89)
- WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (ICLR 2025 · similarity 0.89)
- Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs (ICML 2025 · similarity 0.88)
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.