AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Beyond Drift: Stabilizing Subjective LLM Evaluation with Information-Theoretic Rubrics

Wang Xu, Shan Zhao, Haolan Chen, Tianwei Yan, Chengyu Wang, Shizhao Chen, Qian Wan

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes information-theoretic rubrics to stabilize LLM-as-judge evaluation of subjective tasks by countering dimension drift in free-form evaluation protocols.

Problem

LLM evaluation of subjective tasks (role-playing, humor, emotional intelligence, dialogue quality) suffers from reproducibility problems, as the same evaluator may contradict itself when re-judging the same sample.

Approach

Attributes instability to dimension drift in free-form evaluation protocols such as chain-of-thought reasoning, and proposes information-theoretic rubrics to constrain the evaluation dimensions and stabilize judgments.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use EIG-selected questionnaire instead of CoTLLM judge (GPT-4.1) · Six-benchmark averageself-consistency (Cohen's kappa): 0.6132 -> 0.7786Improved
use EIG-selected questionnaire instead of CheckEval fixed checklistLLM judge (GPT-4.1) · Six-benchmark averageself-consistency (Cohen's kappa): 0.7472 -> 0.7786Improved
use EIG-selected questionnaire instead of CheckEvalLLM judge (GPT-4.1) · Six-benchmark averagehuman alignment (QWK): 0.3118 -> 0.4719Improved
add chain-of-thought to direct scoringLLM Direct · Six-benchmark averageself-consistency (Cohen's kappa): 0.7361 -> 0.6132Hurt
add chain-of-thought to direct scoringLLM Direct · Six-benchmark averagehuman alignment (QWK): 0.3610 -> 0.4270Improved
select questions by EIG instead of randomly (5 questions) (ablation)EIG questionnaire · Dimension coveragecoverage: 60.0% -> 89.17%Improved
select questions by EIG instead of randomly (10 questions) (ablation)EIG questionnaire · Dimension coveragecoverage: 75.0% -> 94.17%Improved
use fixed EIG questionnaire instead of CoT for dimension retention (ablation)LLM judge · six benchmarksdimension exact match: 0.71 -> 1.0Improved
use a different model as question generator (Qwen3-Next-80B + GPT-4.1 vs GPT-4.1 + GPT-4.1) (ablation)EIG pipeline · Heterogeneous pipeline averagehuman alignment (QWK): 0.4810 vs. 0.4806No clear effect
use a different model as question generator (Qwen3-Next-80B + GPT-4.1 vs GPT-4.1 + GPT-4.1) (ablation)EIG pipeline · Heterogeneous pipeline averageself-consistency (Cohen's kappa): 0.7786 vs. 0.7725No clear effect
use EIG pipeline with GPT-4.1 instead of best baseline (CheckEval) (ablation)EIG pipeline · Heterogeneous pipeline averagehuman alignment (QWK): 0.4472 -> 0.4810Improved
use EIG pipeline with Qwen3-Next-80B instead of best baseline (ablation)EIG pipeline · Heterogeneous pipeline averageself-consistency (Cohen's kappa): 0.6592 -> 0.7051Improved

Surprising findings

Experimental setup

Metrics: re-judgment consistency

Baselines: free-form chain-of-thought evaluation

Builds on: chain-of-thought

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy