AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim

ICML 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add detailed evaluation instructions with rubric definitions (ablation)LLM judge prompts · TopicalChatprompt consistency CV: GPT-4o Naturalness 0.28 -> 0.10Improved
add chain-of-thought on top of detailed prompt (ablation)LLM judge prompts · TopicalChat Naturalnessprompt consistency CV: GPT-4o 0.10 -> 0.01Improved
add chain-of-thought on top of detailed prompt (ablation)LLM judge prompts · TopicalChatmarginal reliability rhoNo clear effect
use 5-point instead of 3-point rating scale (ablation)LLM judges · TopicalChat Naturalness/Coherence/Engagingnessmarginal reliability rho: 0.91-0.95 (Naturalness)Improved
use 7-point instead of 5-point rating scale (ablation)LLM judges · TopicalChatmarginal reliability rhoNo clear effect
use larger model (Qwen3-235B vs Qwen3-30B) (ablation)LLM judges · SummEval Relevanceprompt consistency CV: 0.17 -> 0.09Improved
use larger model (GPT-4o vs GPT-4o-mini) (ablation)LLM judges · HelpSteer-2 Helpfulnessmarginal reliability rho: 0.67 -> 0.84Improved
use larger model (Qwen3-VL-235B vs 30B, GPT-4o vs mini) (ablation)VLM judges · VIEScoreCV and rhoNo clear effect
evaluate vision-language judge (VIEScore) instead of text judgesLLM judges · VIEScore vs NLPprompt consistency CV: NLP CV < 0.30 vs VIEScore 0.16 to 1.32Hurt
judge perceptual quality (PQ) with VLMsVLM judges · VIEScore PQdiscrimination breadth ratio: theta ratio 2.03-4.40Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy