Research map / Large language models / Language, safety and interpretability
Multicalibration for Confidence Scoring in LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies multicalibration (IGLB, a new overfitting-resistant variant of iterative grouped binning, plus GCULR) to post-process LLM answer confidence scores for hallucination detection, using clustering and LLM self-annotation to form groups.
Problem
LLM confidence scores for answers are poorly calibrated, and marginal calibration hides large heterogeneity in correctness across prompt types; there are also no explicit group features in LLM data to condition calibration on.
Approach
Use multicalibration with automatically generated groups (embedding clusters and LLM yes/no self-annotations), and reduce overfitting of iterative multicalibration via upper/lower-set bins and linear scaling patches with early stopping.
Key results
- IGLB achieves the lowest MSE on most datasets, e.g. 0.2416 on BigBench vs 0.3242 uncalibrated and 0.2588 for IGHB (StableBeluga/True-False score averages).
- GCULR attains the highest accuracy on several datasets (e.g. 0.7037 on TriviaQA, 0.7779 on MathQA), while IGLB has lowest gASCE on most MMLU topics (e.g. Ethics 0.0189 vs 0.1775 uncalibrated).
- The lower/upper-set binning modification accounts for most of IGLB's gain over IGHB (IGHB^tau MSE nearly matches IGLB), while linear scaling inside standard bins worsens IGHB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · BigBench | MSE: 0.3242 (0.0201) -> 0.2416 (0.0027) | Helped |
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · MMLU | MSE: 0.3045 (0.0315) -> 0.2254 (0.0084) | Helped |
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · OpenBookQA | MSE: 0.2608 (0.0037) -> 0.236 (0.0091) | Helped |
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · TruthfulQA | MSE: 0.4762 (0.135) -> 0.2016 (0.0437) | Helped |
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · MathQA | MSE: 0.3767 (0.0817) -> 0.1727 (0.0047) | Helped |
| Apply IGLB calibration to True/False softmax scores | LLM confidence scores · TriviaQA | MSE: 0.2802 (0.029) -> 0.1974 (0.0308) | Helped |
| Use lower and upper sets as bins (ablation) | IGHB multicalibration · BigBench | MSE: 0.2588 (0.0157) -> 0.2428 (0.0024) | Helped |
| Use lower and upper sets as bins (ablation) | IGHB multicalibration · MMLU | MSE: 0.2517 (0.0138) -> 0.2269 (0.0096) | Helped |
| Use lower and upper sets as bins (ablation) | IGHB multicalibration · OpenBookQA | MSE: 0.2517 (0.0062) -> 0.2372 (0.0083) | Helped |
| Use lower and upper sets as bins (ablation) | IGHB multicalibration · TruthfulQA | MSE: 0.3051 (0.0147) -> 0.2043 (0.0451) | Helped |
| Use linear scaling patches without changing the bins (ablation) | IGHB multicalibration · BigBench | MSE: 0.2588 (0.0157) -> 0.2521 (0.0101) | Hurt |
| Use linear scaling patches without changing the bins (ablation) | IGHB multicalibration · MMLU | MSE: 0.2517 (0.0138) -> 0.2597 (0.0206) | Hurt |
Surprising findings
- Linear scaling patches applied to standard level-set bins (IGHB^LS) perform worse than plain IGHB, attributed to overfitting.
- GCULR achieves higher accuracy than IGLB on several datasets despite lacking multicalibration guarantees.
Limitations
- Calibration guarantees hold only for prompts distributed like the calibration set, not adversarial prompts.
- Annotation-based grouping has higher computational cost and latency at deployment and quality depends on the LLM.
- Labels for TriviaQA are derived from word-overlap with answer keys, a noisy proxy for correctness.
Experimental setup
Datasets: BigBench MMLU OpenBookQA TruthfulQA MathQA TriviaQA
Benchmarks: BigBench MMLU OpenBookQA TruthfulQA MathQA TriviaQA
Metrics: MSE accuracy gASCE calibration plots
Baselines: Histogram Binning (HB) Linear Scaling (LS) GCULR IGHB uncalibrated scores
Builds on: multicalibration histogram binning platt scaling group-conditional unbiasedness (multi-accuracy)
Closest papers by meaning
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.90)
- SteerConf: Steering LLMs for Confidence Elicitation (NeurIPS 2025 · similarity 0.90)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
- Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.89)
- Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization (NeurIPS 2025 · similarity 0.89)
- Calibrating LLMs with Semantic-level Reward (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.