Research map / Large language models / Reasoning and chain of thought

Calibrating LLMs with Semantic-level Reward

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Calibration with Semantic Reward (CSR), an RL fine-tuning method that couples a correctness reward with a pairwise semantic-agreement calibration reward so that semantic entropy over sampled rollouts becomes a calibrated confidence signal, without verbalized confidence.

Problem

LLMs trained with binary-correctness RLVR become overconfident, and existing verbalized-confidence calibration methods calibrate an unstable reporting interface rather than the model's underlying uncertainty over answer meanings.

Approach

A per-rollout semantic calibration reward that penalizes mismatch between a rollout's pairwise semantic agreement with other rollouts and its correctness, combined with the correctness reward through a linearly scheduled weight and optimized with GRPO; confidence is then read post hoc via semantic entropy.

Key results

• On Llama, CSR reaches average ECE 0.0557 and AUROC 0.8076 across four datasets, versus 0.1072 / 0.7518 for Base and 0.1173 / 0.7662 for RD.
• CSR cuts average ECE to 0.0616 on Qwen (Base 0.0947) and 0.0474 on Mistral (Base 0.0851), with token cost comparable to Base and RLVR.
• Using a lexical F1 threshold instead of the LLM judge keeps most gains (Llama CSR-F1 average ECE 0.1157 vs Base-F1 0.1416), though it is less robust than the LLM judge.

Surprising findings

• The calibration reward alone reaches very high AUROC but collapses accuracy, so the correctness term is necessary.
• RD-trained Mistral checkpoints drift to emitting decimal confidences, causing the RD parsing rate to drop to 0.000 without a fallback parser.

Limitations

• The semantic equivalence judge requires extra LLM calls among rollouts, adding overhead.
• The F1-threshold variant is less robust than the LLM judge, with larger ECE gaps when answers differ in surface form.
• Accuracy on MSMARCO and NQ-Open is mixed across methods and CSR is sometimes lower than Base in accuracy.

Experimental setup

Datasets: HotpotQA

Benchmarks: HotpotQA TriviaQA MSMARCO NQ-Open

Metrics: accuracy ECE AUROC token cost

Baselines: Base RLVR Rewarding Doubt (RD) RLCR

Builds on: GRPO RLVR semantic entropy RLCR Rewarding Doubt

Closest papers by meaning

• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.91)
• Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)
• Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.89)
• Multicalibration for Confidence Scoring in LLMs (ICML 2024 · similarity 0.89)
• A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.89)
• Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator (NeurIPS 2025 · similarity 0.89)