Research map / Large language models / Language, safety and interpretability
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CALM, an automated attack-and-detect framework that quantifies 12 biases in LLM-as-a-Judge using principle-guided LLM perturbations and robustness/consistency rates across six judge models.
Problem
LLM-as-a-Judge is widely used for evaluation and as reward signals, but its reliability is threatened by many under-explored biases, and existing bias studies are narrow and rely on costly human evaluation.
Approach
An automated bias quantification framework that perturbs responses or system instructions with GPT-4o/GPT-4-Turbo guided by principle-based prompts to inject 12 bias types, then measures whether judge decisions remain consistent (robustness rate, consistency rate, accuracy, error rates).
Key results
- Claude-3.5 shows the greatest overall resilience to biases, but even top models show notable weaknesses; e.g., GPT-4-Turbo is inconsistent on sentiment bias (RR 0.653) while ChatGPT scores 0.804.
- Position bias robustness for ChatGPT is 0.566 and most models fall below 0.5 when evaluating three or four answers.
- CoT improves judge accuracy for most models, e.g., GLM-4 gains 7% while GPT-4-Turbo gains only 0.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of answer candidates from two to three or four | LLM judges · Position-bias evaluation | RR: below 0.5 | Hurt |
| Lengthen lower-quality answers without improving their quality | LLM judges · Fact-related dataset — Verbosity | RR | Hurt |
| Vary the stated percentage of people favoring an answer | LLM judges · Alignment dataset — Bandwagon | RR | No clear effect |
| Add irrelevant content to responses | LLM judges · Alignment dataset — Distraction | RR | Hurt |
| Add fake references in URL format rather than quote or book format | LLM judges · Alignment dataset — Authority | RR | Helped |
| Add emotional wording to higher-quality answers | LLM judges · Fact-related dataset — Sentiment | Accuracy and RR | Hurt |
| Add similar emotional wording to lower-quality answers | LLM judges · Fact-related dataset — Sentiment | Accuracy and RR | Helped |
| Add fear-related emotional wording rather than cheerful wording | LLM judges · Fact-related dataset — Sentiment | Accuracy and RR | Hurt |
| Add identity information, especially female or refugee status, to the system prompt | Most tested LLM judges · Alignment dataset — Diversity | RR | Hurt |
| Ask judges to reason step by step before comparing answers | LLM judges · Alignment dataset — CoT | Accuracy: +7% | Helped |
| Ask judges to reason step by step before comparing answers | GPT-4-Turbo · Alignment dataset — CoT | Accuracy: 0.7% | Helped |
| Use the same model to generate and judge answers | LLM judges · Self-enhancement evaluation | ErrorRate_SE | Hurt |
Surprising findings
- Advanced models do not always show better bias robustness; the most advanced model is not always the most reliable judge.
- Self-enhancement error rate is high for Qwen2 (16.1) and Claude-3.5 (7.48) while GPT-4-Turbo and GLM-4 are lower.
Limitations
- Relies on GPT-4o/GPT-4-Turbo to generate perturbations, which may introduce its own artifacts.
- Some datasets contain NSFW content; the paper warns of offensive material.
- Bias types and datasets are limited in size (150-500 samples per bias).
Experimental setup
Datasets: Truthy-DPO-v0.1 Emerton-DPO-Pairs-Judge Orca-DPO-Pairs Py-DPO-v0.1 Roleplay-NSFW GSM8K MATH ScienceQA CommonsenseQA Quora-QuAD TruthfulQA
Benchmarks: CALM (fact-related, alignment, refinement-aware datasets)
Metrics: robustness rate consistency rate accuracy error rate
Baselines: ChatGPT GPT-4-Turbo GPT-4o Claude-3.5 GLM-4 Qwen2-72B-Instruct
Builds on: constitutional AI DPO datasets MT-Bench
Closest papers by meaning
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.91)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.91)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.90)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.89)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.89)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.