Research map / Large language models / Language, safety and interpretability
QA-Calibration of Language Model Confidence Scores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QA-calibration, which conditions calibration on kd-tree groups of question-answer pairs, and proposes posthoc QA binning and scaling-QA-binning methods with distribution-free guarantees, validated on verbalized confidence from Mistral and Gemma across five QA benchmarks.
Problem
Average-case calibration of LM confidence scores in generative QA can be miscalibrated for specific user-relevant groups of question-answer pairs, so it is hard to interpret for decision-making.
Approach
Defines QA-calibration as calibration conditional on a fixed mapping beta over QA pairs (instantiated with DistilBERT embeddings and kd-tree partitions), and posthoc calibrates per partition using UMD histogram binning, optionally preceded by hierarchical (partially pooled) logistic scaling to pool information across partitions.
Key results
- On MMLU with Mistral and Verb1S-Top1, HS-QAB reduces CE(h;beta) to 0.149 versus 0.639 for uncalibrated elicited confidence (None).
- Across datasets, QA-calibration methods achieve the lowest CE(h;beta) in most settings, with the paper reporting roughly 10-40% improvement in QA-calibration performance.
- Scaling QA binning variants reach AUAC up to about 0.54 on TriviaQA (Mistral, Ling1S-Top1 with S-QAB), exceeding None's 0.537 in that setting, while the paper reports up to 30% selective answering gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply QA-calibration posthoc instead of leaving elicited confidence scores uncalibrated | Mistral · MMLU, Ling1S-Top1 | QA-calibration error: 0.532 ± 0.008 -> 0.16 ± 0.005 | Helped |
| Apply hierarchical scaling QA binning instead of no recalibration | Mistral · MMLU, Verb1S-Top1 | AUAC: 0.255 ± 0.008 -> 0.306 ± 0.035 | Helped |
| Use QA binning instead of scaling-binning | Mistral · SciQ, Ling1S-Top1 | QA-calibration error: 0.466 ± 0.003 -> 0.165 ± 0.003 | Helped |
| Use QA-calibration-aware scaling-binning instead of standard scaling-binning | Mistral · TriviaQA, Ling1S-Top1 | QA-calibration error: 0.441 ± 0.003 -> 0.184 ± 0.005 | Helped |
| Apply hierarchical scaling QA binning instead of no recalibration | Mistral · OpenBookQA, Ling1S-Top1 | AUAC: 0.38 ± 0.015 -> 0.37 ± 0.065 | No clear effect |
| Use QA-partition-aware calibration rather than standard average-case calibration | Posthoc calibration methods · QA benchmarks | QA-calibration error | Helped |
| Use hierarchical scaling rather than fully pooled scaling before QA binning | Scaling QA binning · QA benchmarks | QA-calibration error | Mixed |
Surprising findings
- The kd-tree depth selected by AUAC tuning is never zero, i.e., the optimal partitioning is not average-case calibration.
- Uncalibrated elicited confidence can have high AUAC yet very high QA-calibration error.
Limitations
- Interpretability depends on the choice of beta and kd-tree depth.
- Algorithms assume a fixed output space for beta, which may not hold as generative QA information grows.
- Ground-truth correctness labels are proxies produced by Llama 3.1 judging semantic equivalence, with noted false negatives.
Experimental setup
Datasets: TriviaQA SciQ BigBench OpenBookQA MMLU
Benchmarks: TriviaQA SciQ BigBench OpenBookQA MMLU
Metrics: QA-calibration error CE(h;beta) AUAC
Baselines: None (elicited confidence) UMD histogram binning Platt scaling scaling-binning
Builds on: histogram binning (UMD) scaling-binning Platt scaling verbalized confidence elicitation
Closest papers by meaning
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.92)
- Conformal Linguistic Calibration: Trading-off between Factuality and Specificity (NeurIPS 2025 · similarity 0.89)
- Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.89)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.88)
- Refine and Align: Confidence Calibration Through Multi-Agent Interaction in VQA (AAAI 2026 · similarity 0.88)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.