Research map / Large language models / Language, safety and interpretability
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
Yidong Wang, Yunze Song, Tingyuan Zhu, Xuanwang Zhang, Zhuohao Yu, Hao Chen, Chiyu Song, Qiufeng Wang, Zhen Wu, Xinyu Dai, Yue Zhang, Cunxiang Wang and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace raw 5-point scoring with distribution-sensitive scoring | LLM-as-a-judge · Score-Comparison, Llama-3.1-70B | Conflict Ratio (%): 23.32 -> 14.89 | Improved |
| use likelihood-aware aggregation instead of two-pass swap-order tie rule | pairwise judge · Pairwise transitivity k=5, Llama-3.2-3B | NTR (%): 54.69 -> 17.76 | Improved |
| add likelihood-aware aggregation to pairwise baseline (ablation) | pairwise judge · Pairwise transitivity k=4, Llama-3.1-70B | NTR (%): 7.23 -> 1.94 | Improved |
| add softmax normalization to 5-scale scoring (ablation) | single-score judge · Conflict Ratio, Llama-3.1-70B | Conflict Ratio (%): 23.32 -> 17.08 | Improved |
| add 100-point scale on top of softmax (ablation) | single-score judge · Conflict Ratio | Conflict Ratio (%): 26.10 -> 24.54 (Llama-3.1-8B); 17.08 -> 17.94 (Llama-3.1-70B) | Mixed |
| use PPL-based aggregation instead of likelihood-aware aggregation (ablation) | pairwise judge · NTR k=4 | NTR (%): 3.79 -> 6.56 (Llama-3.1-8B); 6.26 -> 4.80 (GPT-3.5-Turbo) | Mixed |
| apply TrustJudge to reasoning-distilled judge | DeepSeek-R1-Distill-Llama-8B · Conflict Ratio | Conflict Ratio (%): 58.75 -> 49.28 | Improved |
| use TrustJudge pairwise aggregation instead of baseline | pairwise judge · Exact match, Llama-3.1-70B | Exact match (%): 80.42 -> 81.61 | Improved |
| build DPO preference data with 100-scale softmax TrustJudge scores instead of 5-scale baseline | Llama-3.1-8B-SFT · AlpacaEval 2 | LC win rate: 19.13 -> 20.52 | Improved |
| build DPO preference data with 100-scale softmax TrustJudge scores instead of 5-scale baseline | Qwen2.5-7B-SFT · AlpacaEval 2 | LC win rate: 16.82 -> 18.54 | Improved |
| use larger Gemma judge (27B vs 9B) (ablation) | judge model · inconsistency | inconsistency | Hurt |
Closest papers by meaning
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.91)
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.91)
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.90)
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025 · similarity 0.90)
- From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.89)
- Investigating Non-Transitivity in LLM-as-a-Judge (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
