Research map / Large language models / Instruction and fine-tuning

An Empirical Analysis of Uncertainty in Large Language Model Evaluations

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Empirically studies evaluation uncertainty (token-probability confidence) of 9 LLM evaluators on MT-Bench and PandaLM, shows CoT/reference prompting and CoT-trained evaluators raise confidence, and fine-tunes ConfiLM (Llama3-8B) with response-confidence hints that improves OOD judging on a new 220-instance Olympic 2024 set.

Problem

Model-based LLM-as-a-Judge evaluations are unstable, but the role of evaluator and candidate uncertainty in LLM evaluation is largely unexplored, especially under out-of-distribution data.

Approach

Uses token probabilities as internal confidence for both evaluator (evaluation confidence) and candidate responses (response confidence), analyzes how prompting and specialized training affect it, and fine-tunes an evaluator that receives verbalized response confidence as extra input to improve OOD judging.

Key results

• Average evaluation confidence is 0.654 for single-answer grading versus 0.797 for pairwise comparison on MT-Bench across 6 general evaluators.
• ConfiLM reaches F1 0.621 on Olympic 2024, versus 0.582 for Llama3-8B-Instruct-Finetune and the reported gains of 3.9% and 8.5%, with the 3.9% gain described as relative to Finetune, which is the paper's wording.
• Prometheus2-7B achieves evaluation confidence 0.993 on MT-Bench single-answer grading, above the general-LLM average of 0.654, showing CoT-trained evaluators are more confident.

Measured effects reported in the paper

Surprising findings

• Higher general capability does not guarantee more stable evaluation: GPT-4o has low single-answer evaluation confidence (0.417) on MT-Bench.
• Llama2-70B-Instruct evaluating Llama2-7B-Instruct shows unusually high confidence (0.953), suggesting self-preference bias within a model family.

Limitations

• Only single-round evaluation is studied; multi-turn MT-Bench uses first-turn questions only.
• Proprietary models without token probabilities (e.g., Claude) are not covered.
• OOD test set is small (220 instances) and the evaluation confidence-accuracy relation is only analyzed on Olympic 2024.

Experimental setup

Datasets: Alpaca 52K MT-Bench PandaLM test set

Benchmarks: MT-Bench PandaLM test set Olympic 2024

Metrics: evaluation confidence response confidence F1 win/lose/tie rate rating

Baselines: GPT-4o GPT-4o-mini GPT-3.5-Turbo Llama3-70B-Instruct Llama2-70B-Instruct Qwen2-72B-Instruct Prometheus2-7B Prometheus2-bgb-8x7B PandaLM-7B Llama3-8B-Instruct Llama3-8B-Instruct-Finetune

Builds on: MT-Bench PandaLM Prometheus 2 Alpaca 52K

Closest papers by meaning

• Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.93)
• Large Language Models Must Be Taught to Know What They Don’t Know (NeurIPS 2024 · similarity 0.92)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.91)
• LLMEval: A Preliminary Study on How to Evaluate Large Language Models (AAAI 2024 · similarity 0.91)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
• Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (ICML 2025 · similarity 0.90)