AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation

Peng Lai, Zhihao Ou, Yong Wang, Longyue Wang, Jian Yang, Yun Chen, Guanhua Chen

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 8 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
inject BiasScope-discovered biasesLLaMA-3.1-8B-Instruct as judge · JudgeBench overallerror rate (%): 41.7 -> 52.5Hurt
inject BiasScope-discovered biasesQwen2.5-14B-Instruct as judge · JudgeBench overallerror rate (%): 37.7 -> 47.8Hurt
inject BiasScope-discovered biasesQwen2.5-1.5B-Instruct as judge · JudgeBench Codeerror rate (%): 54.5 -> 54.1No clear effect
inject BiasScope-discovered biasesMistral-7B-Instruct-v0.3 as judge · JudgeBench Reasoningerror rate (%): 47.7 -> 47.3No clear effect
use GPT-OSS-120B as teacher instead of none (ablation)LLaMA-3.1-8B-Instruct as judge · JudgeBench overallerror rate (%): 41.7 -> 53.8Hurt
use stronger teacher GPT-OSS-120B instead of GPT-OSS-20B (ablation)Qwen2.5-7B-Instruct as judge · JudgeBench overallerror rate (%): 50.6 -> 56.5Hurt
use Early-Validate instead of Late-Validate (ablation)BiasScope on Qwen2.5-7B · JudgeBenchnumber of validated biases: 21 -> 27Improved
use Early-Validate instead of Late-Validate (ablation)BiasScope on LLaMA-3.1-8B · JudgeBenchnumber of validated biases: 27 -> 29Improved
add DeeperExplain (ablation)BiasScope on Qwen2.5-1.5B-Instruct · JudgeBenchnumber of biases found: 43 -> 48Improved
add DeeperExplain (ablation)BiasScope on Qwen2.5-7B-Instruct · JudgeBenchnumber of biases found: 25 -> 27Improved
increase data from 25% to 100% (ablation)BiasScope on Mistral-7B-Instruct-v0.3 · RM-Benchnumber of biases found: 12 -> 27Improved
truncate perturbed answers to original lengthmulti-bias perturbation on RewardBench · RewardBencherror rate change: +2.2%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy