Research map / Label-efficient and robust learning / Recognition and long-tail

Calibration without Ground Truth

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Confidence calibration objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a label-free post-processing method that Bregman-projects a miscalibrated primary LLM onto the set of predictors mutually calibrated with a better-calibrated base reference, yielding strict worst-case loss reduction.

Problem

Improving a strong but miscalibrated model's probabilistic predictions without ground-truth labels, using a weaker but better-calibrated reference model.

Approach

Strict improvement under any proper loss is possible iff the primary and reference predictors are not mutually calibrated; the method projects the primary predictor (Bregman divergence from the proper loss) onto the convex set of reference-compatible transformations, guaranteeing worst-case loss reduction equal to the projection distance. Links this to no-arbitrage and no-trade results from econom

Key results

• On CommonsenseQA with Ministral-3-8B, the label-free method reaches ECE 0.0295 vs 0.1682 for the Instruct model, and Brier 0.1777 vs 0.2041.
• On MMLU-Redux with Ministral-3-8B, accuracy rises from 79.21% (Instruct) to 80.52% with the label-free method.
• On LLaMA-3.1-8B MMLU-Redux, the label-free method reduces ECE from 0.2417 (Instruct) to 0.1679, but temperature scaling reaches 0.0352 on the same benchmark.

Surprising findings

• Label-free projection sometimes gives a worse ECE than the supervised temperature-scaling baseline on MMLU-Redux (e.g., Qwen3-8B 0.0659 vs 0.0584 is close, LLaMA-3.1-8B 0.1679 vs 0.0352 is far), indicating gaps remain.

Limitations

• Assumes the reference is well-calibrated; miscalibration of the reference can weaken guarantees.
• Requires a batch of unlabeled data to estimate the joint distribution, which is hard for rare events.
• No universal transformation is optimal across all proper losses; experiments limited to multiple-choice QA.

Experimental setup

Datasets: MMLU-Redux CommonsenseQA

Benchmarks: MMLU-Redux CommonsenseQA

Metrics: accuracy Brier score expected calibration error confidence loss

Baselines: Temperature Scaling

Builds on: temperature scaling weak-to-strong generalization no-trade theorem Bregman divergence from proper scoring rules

Closest papers by meaning

• Truthfulness of Calibration Measures (NeurIPS 2024 · similarity 0.85)
• Testable and Actionable Calibration for Full Swap Regret (NeurIPS 2026 · similarity 0.83)
• Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.83)
• High-Dimensional Calibration from Swap Regret (NeurIPS 2025 · similarity 0.83)
• Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.82)
• Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity (NeurIPS 2025 · similarity 0.82)