Research map / Large language models / Reasoning and chain of thought

Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a calibrated Bradley-Terry-Davidson aggregator over three-way vote counts (win/tie/loss) from repeated thinking-LLM judge samples, fit by DRPS and decided by an MAE-optimal Bayes rule, outperforming self-consistency variants on WMT23 and RewardBench 2.

Problem

Single-sample thinking-LLM judgments for pairwise preferences are noisy, and common aggregation rules (majority vote, soft self-consistency, LLM-based self-aggregation) handle ties poorly and ignore evidential strength when aggregating multiple samples.

Approach

Model the three-way vote counts with a Davidson-style multinomial logit using a polarity feature (log ratio of non-tie votes) and a tie-evidence feature, fit the parameters by minimizing discrete ranked probability score on a small (~5%) calibration set, then output the action minimizing expected MAE.

Key results

• With gemini-2.5-flash at n=12, the calibrated BTD aggregator reaches MAE 0.588 on WMT en-de versus 0.648 for self-consistency, and 0.281-0.287 range on several RB2 tasks versus roughly 0.40 for SC.
• On RB2-Ties at n=12, MAE drops to 0.081 versus 0.158 for SC, and pairwise accuracy reaches 0.918 versus 0.842.
• In leave-one-out comparison on WMT zh-en, the LLM judge with n=12 beats 7 of 8 individual human raters in pairwise accuracy.

Surprising findings

• Forced-choice judging shows substantial positional bias (gemini-2.5-flash 14.5%), which is largely removed when a tie option is allowed.
• Calibrating on RB2-Factuality, which has a 53% tie rate while the LLM predicts ties only about 6% of the time, transfers poorly to other tasks.

Limitations

• Transfer across tasks is inconsistent; calibration on some tasks (e.g., Factuality) hurts others, and cross-family transfer between WMT and RB2 is weak.
• Evaluated only with three thinking LLM judges and pairwise three-way labels; extension to Likert scales and multi-judge settings is left to future work.
• Human-consensus ground truth for WMT is majority vote over noisy ratings, and comparisons use a fixed 5% calibration split with significance tests based on resampling.

Experimental setup

Datasets: WMT23 zh-en WMT23 en-de RewardBench 2 Factuality RewardBench 2 Focus RewardBench 2 Math RewardBench 2 Precise IF RewardBench 2 Safety RewardBench 2 Ties

Benchmarks: WMT23 RewardBench 2

Metrics: MAE pairwise accuracy

Baselines: Greedy decoding Few-shot Self-Consistency Soft Self-Consistency Confidence-Informed Self-Consistency Generative Self-Aggregation Universal Self-Consistency

Builds on: Self-Consistency Bradley-Terry-Davidson Universal Self-Consistency Generative Self-Aggregation Confidence-Informed Self-Consistency

Closest papers by meaning

• Validating LLM-as-a-Judge Systems under Rating Indeterminacy (NeurIPS 2025 · similarity 0.88)
• Investigating Non-Transitivity in LLM-as-a-Judge (ICML 2025 · similarity 0.87)
• Optimal Self-Consistency for Efficient Reasoning with Large Language Models (ICML 2026 · similarity 0.87)
• Bridging Human and LLM Judgments: Understanding and Narrowing the Gap (NeurIPS 2025 · similarity 0.87)
• JudgeBench: A Benchmark for Evaluating LLM-Based Judges (ICLR 2025 · similarity 0.87)
• From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation (NeurIPS 2026 · similarity 0.87)