Research map / Large language models / Retrieval and knowledge

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sem-ECE, a black-box calibration evaluation framework for open-ended QA that samples answers, clusters them into semantic classes with an LLM judge, and uses semantic frequency as confidence via same-sample (Sem1) or held-out (Sem2) estimators.

Problem

Evaluating LLM calibration in open-ended QA where logits are unavailable, verbalized confidence is overconfident, and existing sampling-based methods lack a finite-sample statistical target.

Approach

Use semantic-class frequencies from repeated samples as confidence, and decouple answer selection from confidence evaluation with a held-out block (Sem2-ECE), which removes the winner's-curse upward bias of the same-sample estimator; prove asymptotic unbiasedness and characterize when Sem2 is closer to the oracle ECE.

Key results

• Sem2-ECE is no larger than Sem1-ECE on all 15 model-benchmark pairs and lowest among the three sources in 12 of 15 pairs versus verbalized confidence.
• On HLE, Sem2-ECE is the best confidence source for all five providers.
• Leading-order prediction phi(m*)/sqrt(n) recovers the empirical Sem1-Sem2 ECE gap within 11-27% at regime boundaries with no fitted constants.

Surprising findings

• Verbalized confidence beats Sem-ECE in 3 of 15 pairs, where models show high agreement but low accuracy (e.g., Anthropic SimpleQA mean Sem1 confidence 0.835 vs accuracy 0.482).

Limitations

• Relies on an LLM judge (gpt-5.4) for semantic clustering and correctness grading.
• Theory assumes asymptotic regimes and a leading-order CLT approximation; residuals of 11-27% remain.
• Agreement-based confidence does not measure correctness, so a model can be consistently wrong.

Experimental setup

Datasets: SimpleQA HLE PopQA

Benchmarks: SimpleQA HLE PopQA

Metrics: expected calibration error (L=10 bins) confidence gap reliability diagrams

Baselines: verbalized confidence Sem1-ECE (same-sample self-consistency)

Builds on: expected calibration error semantic uncertainty verbalized confidence

Closest papers by meaning

• QA-Calibration of Language Model Confidence Scores (ICLR 2025 · similarity 0.92)
• SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs (ICLR 2024 · similarity 0.91)
• RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.90)
• Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.90)
• Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.90)
• Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.89)