AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Quantifying Biases in LLM-as-a-Judge Evaluations

Magda Dubois, Harry Coppock, Mario Giulianelli, Ole Kristian Jorgensen, Timo Flesch, Lennart Luettgau, Cozmin Ududec

ICML 2026 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
remove estimated grader severity (beta1) from the linear predictor and recompute agreement (counterfactual) (ablation)Bayesian GLM framework · Simulated autograder vs human scoresKrippendorff alpha: -0.2 -> approximately 0.7Improved
add grader-specific length-difference term to pairwise GLM (ablation)Pairwise Bayesian GLM · Simulated pairwise preferencesmodel fit (LOO/WAIC)Improved
add grader-LLM interaction term to detect self-bias (ablation)Bayesian GLM framework · Simulated scoresmodel fit (LOO/WAIC)Improved
use hierarchical (partial pooling) instead of flat model for grader groups (ablation)Bayesian GLM framework · Simulated multi-grader scoresmodel fit (LOO/WAIC)No clear effect
GPT-4 judge versus human judge on MT-BenchLLM-as-a-judge · MT-Bench pairwiselength bias coefficient: 0.25 -> 1.09Hurt
GPT-4 judging its own outputs versus Llama outputsGPT-4 judge · MT-Bench pairwisesystematic grader shift: 2.20 for GPT-4, -2.91 for LlamaHurt
model comparison across M1-M7 specifications on MT-Bench (ablation)Bayesian GLM framework · MT-BenchLOO/WAICImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy