AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them

Yidong Wang, Yunze Song, Tingyuan Zhu, Xuanwang Zhang, Zhuohao Yu, Hao Chen, Chiyu Song, Qiufeng Wang, Zhen Wu, Xinyu Dai, Yue Zhang, Cunxiang Wang and 2 more

ICLR 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 28 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace raw 5-point scoring with distribution-sensitive scoringLLM-as-a-judge · Score-Comparison, Llama-3.1-70BConflict Ratio (%): 23.32 -> 14.89Improved
use likelihood-aware aggregation instead of two-pass swap-order tie rulepairwise judge · Pairwise transitivity k=5, Llama-3.2-3BNTR (%): 54.69 -> 17.76Improved
add likelihood-aware aggregation to pairwise baseline (ablation)pairwise judge · Pairwise transitivity k=4, Llama-3.1-70BNTR (%): 7.23 -> 1.94Improved
add softmax normalization to 5-scale scoring (ablation)single-score judge · Conflict Ratio, Llama-3.1-70BConflict Ratio (%): 23.32 -> 17.08Improved
add 100-point scale on top of softmax (ablation)single-score judge · Conflict RatioConflict Ratio (%): 26.10 -> 24.54 (Llama-3.1-8B); 17.08 -> 17.94 (Llama-3.1-70B)Mixed
use PPL-based aggregation instead of likelihood-aware aggregation (ablation)pairwise judge · NTR k=4NTR (%): 3.79 -> 6.56 (Llama-3.1-8B); 6.26 -> 4.80 (GPT-3.5-Turbo)Mixed
apply TrustJudge to reasoning-distilled judgeDeepSeek-R1-Distill-Llama-8B · Conflict RatioConflict Ratio (%): 58.75 -> 49.28Improved
use TrustJudge pairwise aggregation instead of baselinepairwise judge · Exact match, Llama-3.1-70BExact match (%): 80.42 -> 81.61Improved
build DPO preference data with 100-scale softmax TrustJudge scores instead of 5-scale baselineLlama-3.1-8B-SFT · AlpacaEval 2LC win rate: 19.13 -> 20.52Improved
build DPO preference data with 100-scale softmax TrustJudge scores instead of 5-scale baselineQwen2.5-7B-SFT · AlpacaEval 2LC win rate: 16.82 -> 18.54Improved
use larger Gemma judge (27B vs 9B) (ablation)judge model · inconsistencyinconsistencyHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy