Research map / Large language models / Language, safety and interpretability
BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
Peng Lai, Zhihao Ou, Yong Wang, Longyue Wang, Jian Yang, Yun Chen, Guanhua Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| inject BiasScope-discovered biases | LLaMA-3.1-8B-Instruct as judge · JudgeBench overall | error rate (%): 41.7 -> 52.5 | Hurt |
| inject BiasScope-discovered biases | Qwen2.5-14B-Instruct as judge · JudgeBench overall | error rate (%): 37.7 -> 47.8 | Hurt |
| inject BiasScope-discovered biases | Qwen2.5-1.5B-Instruct as judge · JudgeBench Code | error rate (%): 54.5 -> 54.1 | No clear effect |
| inject BiasScope-discovered biases | Mistral-7B-Instruct-v0.3 as judge · JudgeBench Reasoning | error rate (%): 47.7 -> 47.3 | No clear effect |
| use GPT-OSS-120B as teacher instead of none (ablation) | LLaMA-3.1-8B-Instruct as judge · JudgeBench overall | error rate (%): 41.7 -> 53.8 | Hurt |
| use stronger teacher GPT-OSS-120B instead of GPT-OSS-20B (ablation) | Qwen2.5-7B-Instruct as judge · JudgeBench overall | error rate (%): 50.6 -> 56.5 | Hurt |
| use Early-Validate instead of Late-Validate (ablation) | BiasScope on Qwen2.5-7B · JudgeBench | number of validated biases: 21 -> 27 | Improved |
| use Early-Validate instead of Late-Validate (ablation) | BiasScope on LLaMA-3.1-8B · JudgeBench | number of validated biases: 27 -> 29 | Improved |
| add DeeperExplain (ablation) | BiasScope on Qwen2.5-1.5B-Instruct · JudgeBench | number of biases found: 43 -> 48 | Improved |
| add DeeperExplain (ablation) | BiasScope on Qwen2.5-7B-Instruct · JudgeBench | number of biases found: 25 -> 27 | Improved |
| increase data from 25% to 100% (ablation) | BiasScope on Mistral-7B-Instruct-v0.3 · RM-Bench | number of biases found: 12 -> 27 | Improved |
| truncate perturbed answers to original length | multi-bias perturbation on RewardBench · RewardBench | error rate change: +2.2% | Improved |
Closest papers by meaning
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.93)
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (NeurIPS 2025 · similarity 0.91)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.91)
- Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data (ICLR 2025 · similarity 0.89)
- Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators (ICML 2025 · similarity 0.87)
- Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
