Research map / Large language models / Reasoning and chain of thought

Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DCPO, which decouples reasoning and confidence in RLVR by verbalizing confidence after reasoning, using separate rewards and masked gradients, with a hybrid group- and instance-level calibration target, to reduce RLVR-induced over-confidence without losing accuracy.

Problem

RLVR training makes LLMs over-confident on incorrect answers (calibration degeneration), and existing calibration-aware RL methods that couple calibration into the correctness objective show an accuracy-calibration tradeoff.

Approach

Split each response into a reasoning block and a verbalized confidence block, assign accuracy reward to the reasoning tokens and a calibration reward to confidence tokens via masked gradients, and supervise confidence with a hybrid of group-level mean correctness and instance-level correctness from GRPO rollouts.

Key results

• On the five math benchmarks averaged, DCPO reaches 60.8% accuracy (vs 60.4% GRPO with logits confidence) and ECE 0.128 (vs 0.372 for GRPO verbal), with AUROC 0.881.
• On AIME24, DCPO keeps accuracy at 41.6% (GRPO 40.0% verbal) while cutting PCE from 0.510 to 0.212.
• On code benchmarks, DCPO averages ECE 0.175 vs 0.287 for GRPO while achieving the highest average accuracy (0.515 vs 0.514).

Surprising findings

• Post-hoc confidence classifier on GRPO reduces ECE only marginally and has low AUROC (0.642), suggesting RLVR distorts internal representations.
• RLCR and CCGSPG reduce calibration error at a clear accuracy cost, e.g. RLCR AIME24 accuracy falls from 40.0% to 32.8%.

Limitations

• Theoretical results rely on simplifying assumptions (i.i.d. Bernoulli correctness, Fisher-metric gradient conflict under stated conditions).
• Experiments use one main model (Qwen3-8B) with a few additional models; results use single training runs with small evaluation sets (AIME/AMC) where variance is high.
• The lambda hyperparameter and structured-output format require tuning; format penalty needed.

Experimental setup

Datasets: DeepScaleR PrimeIntellect-verifier

Benchmarks: MATH-500 AIME24 AIME25 AMC23 AMC24 LiveCodeBench v5 LiveCodeBench v6 HumanEval+ MATH Llama3.1 and Qwen3-14B generalization set

Metrics: accuracy ECE PCE AUROC

Baselines: GRPO ConfClass RLCR CCGSPG

Builds on: GRPO RLCR CCGSPG

Closest papers by meaning

• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.91)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.90)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.90)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (ICLR 2026 · similarity 0.90)
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding (ICLR 2026 · similarity 0.90)
• Taming Overconfidence in LLMs: Reward Calibration in RLHF (ICLR 2025 · similarity 0.90)