Research map / Large language models / Reasoning and chain of thought
Beyond Drift: Stabilizing Subjective LLM Evaluation with Information-Theoretic Rubrics
Wang Xu, Shan Zhao, Haolan Chen, Tianwei Yan, Chengyu Wang, Shizhao Chen, Qian Wan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes information-theoretic rubrics to stabilize LLM-as-judge evaluation of subjective tasks by countering dimension drift in free-form evaluation protocols.
Problem
LLM evaluation of subjective tasks (role-playing, humor, emotional intelligence, dialogue quality) suffers from reproducibility problems, as the same evaluator may contradict itself when re-judging the same sample.
Approach
Attributes instability to dimension drift in free-form evaluation protocols such as chain-of-thought reasoning, and proposes information-theoretic rubrics to constrain the evaluation dimensions and stabilize judgments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EIG-selected questionnaire instead of CoT | LLM judge (GPT-4.1) · Six-benchmark average | self-consistency (Cohen's kappa): 0.6132 -> 0.7786 | Improved |
| use EIG-selected questionnaire instead of CheckEval fixed checklist | LLM judge (GPT-4.1) · Six-benchmark average | self-consistency (Cohen's kappa): 0.7472 -> 0.7786 | Improved |
| use EIG-selected questionnaire instead of CheckEval | LLM judge (GPT-4.1) · Six-benchmark average | human alignment (QWK): 0.3118 -> 0.4719 | Improved |
| add chain-of-thought to direct scoring | LLM Direct · Six-benchmark average | self-consistency (Cohen's kappa): 0.7361 -> 0.6132 | Hurt |
| add chain-of-thought to direct scoring | LLM Direct · Six-benchmark average | human alignment (QWK): 0.3610 -> 0.4270 | Improved |
| select questions by EIG instead of randomly (5 questions) (ablation) | EIG questionnaire · Dimension coverage | coverage: 60.0% -> 89.17% | Improved |
| select questions by EIG instead of randomly (10 questions) (ablation) | EIG questionnaire · Dimension coverage | coverage: 75.0% -> 94.17% | Improved |
| use fixed EIG questionnaire instead of CoT for dimension retention (ablation) | LLM judge · six benchmarks | dimension exact match: 0.71 -> 1.0 | Improved |
| use a different model as question generator (Qwen3-Next-80B + GPT-4.1 vs GPT-4.1 + GPT-4.1) (ablation) | EIG pipeline · Heterogeneous pipeline average | human alignment (QWK): 0.4810 vs. 0.4806 | No clear effect |
| use a different model as question generator (Qwen3-Next-80B + GPT-4.1 vs GPT-4.1 + GPT-4.1) (ablation) | EIG pipeline · Heterogeneous pipeline average | self-consistency (Cohen's kappa): 0.7786 vs. 0.7725 | No clear effect |
| use EIG pipeline with GPT-4.1 instead of best baseline (CheckEval) (ablation) | EIG pipeline · Heterogeneous pipeline average | human alignment (QWK): 0.4472 -> 0.4810 | Improved |
| use EIG pipeline with Qwen3-Next-80B instead of best baseline (ablation) | EIG pipeline · Heterogeneous pipeline average | self-consistency (Cohen's kappa): 0.6592 -> 0.7051 | Improved |
Surprising findings
- Even the same evaluator may contradict itself when re-judging the exact same sample.
Experimental setup
Metrics: re-judgment consistency
Baselines: free-form chain-of-thought evaluation
Builds on: chain-of-thought
Closest papers by meaning
- Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation (ICLR 2025 · similarity 0.88)
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.88)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.87)
- Accelerating Unbiased LLM Evaluation via Synthetic Feedback (ICML 2025 · similarity 0.87)
- Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models (ICLR 2024 · similarity 0.87)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
