Research map / Large language models / Instruction and fine-tuning
Calibrating Large Language Models with Sample Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives LLM confidence from the distribution of multiple sampled generations using agreement, entropy, and first-second-distance consistency measures, and shows they beat post-hoc verbalized and probabilistic calibration baselines across nine reasoning datasets.
Problem
LLM predictions are poorly calibrated, and conventional calibration methods are costly or infeasible for large proprietary models lacking logits or training data.
Approach
Measure confidence as the consistency of multiple randomly sampled generations, extending agreement-based consistency with an entropy-based measure and a first-second-distance (FSD) measure that captures the full answer distribution.
Key results
- Consistency-based metrics achieve lower Brier Scores than the best baseline on average, e.g., GPT-4 entropy/FSD 0.114-0.116 vs. 0.154 for best verbalized baseline.
- For open-source models, agreement gives LLaMA-70B Brier 0.154 vs. 0.252 for raw logit.
- Explanation-based prompting (CoT, FCoT) improves calibration and scaling from 7B to 70B widens the gap; instruction-tuned Mistral-7B-instruct is worse calibrated than base Mistral-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sample-consistency calibration (agreement, entropy, or FSD) instead of post-hoc baselines | LLMs on reasoning tasks · Nine reasoning datasets, averaged across domains | Brier Score | Helped |
| Prompt the model to generate an explanation before its answer | LLMs using consistency calibration · Reasoning tasks | Brier Score | Helped |
| Scale LLaMA model size upward | LLaMA · Reasoning tasks | Brier Score | Helped |
| Apply instruction-tuning | Mistral-7B · Reasoning tasks | Brier Score | Hurt |
| Increase the number of sampled generations | GPT-3.5-turbo with CoT prompting · Four datasets | Brier Score: 3-5 samples; 15-20 samples | Helped |
| Use FCoT instead of standard prompting | GPT-4 · All datasets, averaged accuracy | accuracy: 79.3 -> 90.9 | Helped |
| Use CoT instead of standard prompting | LLaMA-13B · All datasets, averaged accuracy | accuracy: 43.2 -> 66.8 | Helped |
Surprising findings
- Instruction-tuning worsens calibration: base Mistral-7B outperforms Mistral-7B-instruct on nearly all prompting strategies and tasks.
- Verbalized confidence baselines are often worse than simple consistency, and in some cases logit baselines are competitive on individual datasets.
Limitations
- No single consistency metric is universally best across models and datasets.
- Uses n=40 samples at a single temperature (0.4), which is costly; sensitivity to temperature not analyzed.
- Only measures consistency over final answers, ignoring intermediate reasoning steps.
Experimental setup
Datasets: GSM8K SVAMP MultiArith ASDiv StrategyQA Date Understanding Sports Understanding SayCan CLUTRR
Benchmarks: GSM8K SVAMP MultiArith ASDiv StrategyQA Date Understanding Sports Understanding SayCan CLUTRR
Metrics: Brier Score ECE Macro-F1 accuracy
Baselines: raw logit P(True) 0-shot P(True) 8-shot verbalized confidence (linguistic) verbalized confidence (percentage)
Builds on: self-consistency P(True) chain-of-thought Faithful CoT Program of Thoughts
Closest papers by meaning
- Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach (ICML 2025 · similarity 0.92)
- Beware of Calibration Data for Pruning Large Language Models (ICLR 2025 · similarity 0.91)
- Thermometer: Towards Universal Calibration for Large Language Models (ICML 2024 · similarity 0.91)
- Calibrated Language Models and How to Find Them with Label Smoothing (ICML 2025 · similarity 0.91)
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.91)
- Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.