Research map / Large language models / Reasoning and chain of thought

Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RLCR, which adds a bounded proper scoring rule (Brier score) to the binary correctness reward so reasoning models learn to answer accurately and output calibrated verbalized confidence, improving calibration in- and out-of-domain without losing accuracy.

Problem

Binary correctness rewards in RLVR reward guessing and overconfidence, degrading calibration and increasing hallucination in reasoning models, especially out of domain.

Approach

Augment the binary correctness reward with a negative Brier score on a verbalized confidence output; a theorem shows that any bounded proper scoring rule combined this way makes correctness and calibration jointly optimal, while unbounded log loss can make models abstain or output wrong answers.

Key results

• On HotpotQA, RLCR reaches 62.1% accuracy (vs 63.0% RLVR) and cuts ECE from 0.37 (RLVR) to 0.03.
• Out-of-domain averaged over six datasets, RLCR achieves 56.2% accuracy and ECE 0.21, versus 53.9% and 0.46 for RLVR.
• On Math, RLCR gets 72.7% accuracy with ECE 0.10 versus RLVR's 72.9% accuracy and ECE 0.26.

Surprising findings

• Log-score calibration rewards combined with correctness can make the model abstain with confidence 0 and collapse to zero accuracy in a toy task.
• The Answer Probability baseline has high AUROC in-domain (0.72) but poor calibration because the model commits to answers during reasoning.

Limitations

• Out-of-domain calibration error remains high in absolute terms, and models still assign high confidence to contradictory answers.
• SFT warmup on Math reduces out-of-domain accuracy, possibly from catastrophic forgetting.
• Experiments use a single 7B base model and mainly short-answer QA and math tasks.

Experimental setup

Datasets: HotpotQA-Modified HotpotQA distractor Big-Math

Benchmarks: HotpotQA TriviaQA SimpleQA GPQA Math-500 GSM8K CommonsenseQA Big-Math

Metrics: accuracy AUROC Brier score expected calibration error

Baselines: RLVR RLVR + BCE Classifier RLVR + Brier Classifier RLVR + Probe Answer Probability Base model SFT+RLCR

Builds on: GRPO RLVR Brier score Big-Math HotpotQA

Closest papers by meaning

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.91)
• Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.91)
• Calibrating LLMs with Semantic-level Reward (NeurIPS 2026 · similarity 0.91)