Research map / Large language models / Language, safety and interpretability
Quantifying Biases in LLM-as-a-Judge Evaluations
Magda Dubois, Harry Coppock, Mario Giulianelli, Ole Kristian Jorgensen, Timo Flesch, Lennart Luettgau, Cozmin Ududec
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove estimated grader severity (beta1) from the linear predictor and recompute agreement (counterfactual) (ablation) | Bayesian GLM framework · Simulated autograder vs human scores | Krippendorff alpha: -0.2 -> approximately 0.7 | Improved |
| add grader-specific length-difference term to pairwise GLM (ablation) | Pairwise Bayesian GLM · Simulated pairwise preferences | model fit (LOO/WAIC) | Improved |
| add grader-LLM interaction term to detect self-bias (ablation) | Bayesian GLM framework · Simulated scores | model fit (LOO/WAIC) | Improved |
| use hierarchical (partial pooling) instead of flat model for grader groups (ablation) | Bayesian GLM framework · Simulated multi-grader scores | model fit (LOO/WAIC) | No clear effect |
| GPT-4 judge versus human judge on MT-Bench | LLM-as-a-judge · MT-Bench pairwise | length bias coefficient: 0.25 -> 1.09 | Hurt |
| GPT-4 judging its own outputs versus Llama outputs | GPT-4 judge · MT-Bench pairwise | systematic grader shift: 2.20 for GPT-4, -2.91 for Llama | Hurt |
| model comparison across M1-M7 specifications on MT-Bench (ablation) | Bayesian GLM framework · MT-Bench | LOO/WAIC | Improved |
Closest papers by meaning
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.94)
- RocketEval: Efficient automated LLM evaluation via grading checklist (ICLR 2025 · similarity 0.92)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.92)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.91)
- Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.91)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
