Research map / Large language models / Reasoning and chain of thought

Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes correctness-calibrated decoding rules (Greedy-Threshold, Calibrated-TopK, Calibrated-ε) that truncate or greedify low-confidence sampling steps using rank-wise correctness estimated from teacher-forced calibration grids, improving majority-voted reasoning accuracy.

Problem

Existing decoding methods increase exploration at high-entropy steps while post-hoc filters reject low-confidence samples, conflicting in how they treat uncertainty; it is unclear how to balance diversity and per-token accuracy when sampling multiple reasoning chains.

Approach

Treat low-confidence steps as epistemic error-prone states rather than branch points: greedily decode when max probability is low (Greedy-Threshold), and set truncation via a learning-free calibration grid mapping confidence bins and ranks to estimated correctness (Calibrated-TopK) or a log-log linear map from probability to correctness (Calibrated-ε).

Key results

• Calibrated-ε reaches 47.1 maj@32 on GSM8K with Qwen2.5-0.5B-Instruct versus 38.6 for no restrictions, and Calibrated-TopK matches it at 47.1.
• Greedy-Threshold added on top of top-k improves Qwen2.5-1.5B-Instruct GSM8K maj@32 by +2.8 points (73.9 to 76.7).
• On GPT-OSS-20B AIME25, Greedy-Threshold raises maj@32 from 90.0 to 91.1 and overall correctness from 56.1% to 59.9%.

Surprising findings

• Sampling from the lowest-confidence bin yields no majority-voting accuracy gain and little diversity, contrary to the view that high-entropy steps are useful exploration points.
• On MBPP, where pass@k rewards diversity, the calibrated samplers underperform min-p, and Calibrated-ε degrades on Qwen2.5-0.5B-Instruct at higher k.

Limitations

• Calibrated-ε quality depends on the fit of the log-log probability-to-correctness map; noisy calibration (e.g., MBPP) reduces gains, and larger models see diminishing gains.
• Gains shrink for stronger models and at low temperature, where diversity is already reduced; the method underperforms on diversity-driven pass@k code tasks.
• Calibration requires a reference dataset with ground-truth answers (in-domain or OOD proxy), and the creative-writing applicability is not evaluated.

Experimental setup

Datasets: GSM8K MMLU-Pro Big-Bench-Hard alpaca-gpt4-en MBPP AIME24 AIME25

Benchmarks: GSM8K MMLU-Pro Big-Bench-Hard MBPP AIME24 AIME25

Metrics: maj@k pass@k accuracy unique answers

Baselines: top-k top-p min-p EDT epsilon-sampling eta-sampling

Builds on: epsilon-sampling top-k sampling min-p sampling entropy-dependent temperature

Closest papers by meaning

• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.91)
• CITE: Anytime Valid Statistical Inference in LLM Self-Consistency (NeurIPS 2026 · similarity 0.91)
• Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning (AAAI 2026 · similarity 0.90)
• Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution (ICML 2026 · similarity 0.90)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.90)