Research map / Large language models / Language, safety and interpretability

CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CEB, a compositional benchmark (11,004 samples) for LLM social bias evaluation, organized by bias type, social group, and task, with unified metrics across existing and newly constructed datasets.

Problem

Existing LLM bias evaluation datasets cover only narrow bias aspects and use incompatible metrics, making results hard to compare across datasets and models.

Approach

A compositional taxonomy (bias type x social group x task) that assigns each dataset a configuration, enabling unified metrics across datasets and construction of new datasets (CEB) filling uncovered configurations, with GPT-4 used to generate and augment samples.

Key results

• GPT-4 and GPT-3.5 achieve the best Recognition/Selection micro F1 on most CEB configurations; e.g., GPT-4 reaches 100.0 on CEB-Selection-T across all four social groups.
• Almost all LLMs perform poorly on CEB-Recognition-S (near 50% F1), while toxicity datasets are easier than stereotyping datasets.
• Human ratings of bias in continuation/conversation outputs are generally aligned with GPT-4 scores, with human scores slightly lower than GPT-4's.

Measured effects reported in the paper

Surprising findings

• GPT models do not outperform smaller open models on stereotyping in Continuation and Conversation tasks, despite dominating Recognition and Selection.
• Llama2 and Llama3 show very high refusal-to-answer rates on Selection tasks, while Mistral-7b shows near-zero RtA.

Limitations

• Only four social groups (age, gender, race, religion) and two bias types (stereotyping, toxicity) are covered; physical appearance and other groups are left to future work.
• Only 100 samples evaluated per configuration, and much of the data is GPT-4-generated or augmented.
• Indirect evaluation relies on GPT-4 and Perspective API as judges, with human evaluation limited to a small sample.

Experimental setup

Datasets: BBQ HolisticBias PANDA WinoBias StereoSet RedditBias CrowS-Pairs BOLD Adult Credit Jigsaw

Benchmarks: CEB WinoBias StereoSet RedditBias CrowS-Pairs

Metrics: micro F1 RtA rate GPT-4 bias score Perspective API toxicity score Demographic Parity Equalized Odds Unfairness Score

Baselines: GPT-3.5 GPT-4 Llama2-7b Llama2-13b Llama3-8b Mistral-7b

Builds on: BBQ HolisticBias StereoSet CrowS-Pairs

Closest papers by meaning

• Mitigating Social Bias in Large Language Models: A Multi-Objective Approach Within a Multi-Agent Framework (AAAI 2025 · similarity 0.88)
• Investigating and Mitigating Undesirable Biases in Large Language Models (AAAI 2025 · similarity 0.88)
• Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.87)
• LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.87)
• Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
• RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.86)