Research map / Large language models / Reasoning and chain of thought

Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a conformal, inference-time abstention framework for self-consistency CoT that replaces majority voting with score-weighted voting and calibrates a vote-confidence threshold via conformal risk control to bound the confident-error rate.

Problem

Self-consistency CoT aggregates sampled reasoning paths by majority vote, which can confidently return a wrong consensus; the paper targets aggregation-level uncertainty and when a system should abstain instead of answering.

Approach

Score-weighted vote share is used as an aggregation confidence; an abstention threshold on it is calibrated with conformal risk control to guarantee a finite-sample bound on the confident-error rate. Score separability is identified as the condition under which abstention provably improves selective accuracy, with a closed-form accuracy-gain predictor computed from calibration data.

Key results

• On GSM8K the method reaches 90.1% selective accuracy while abstaining on less than 5% of problems, versus 82% for majority voting.
• Realized confident-error rates are reported as consistent with the targets across ten dataset-model configurations, four models, and three score families.
• The closed-form accuracy predictor from calibration data closely tracks observed held-out selective accuracy; gains shrink on HotpotQA where separability is low.

Surprising findings

• Perplexity (SC) scores underperform across the board because of their low separability gap, so score choice rather than the guarantee drives utility.

Limitations

• Confident-error guarantee is marginal, not per-input.
• Requires exchangeability; no guarantee under distribution shift.
• Evaluated only on tasks with extractable closed-form answers.

Experimental setup

Datasets: GSM8K MATH MATH-Hard HotpotQA

Benchmarks: GSM8K MATH MATH-Hard HotpotQA

Metrics: selective accuracy confident-error rate yield accuracy-yield frontier AUC

Baselines: majority voting score-weighted voting best-of-m

Builds on: conformal risk control self-consistency confidence-weighted self-consistency Youden J-statistic

Closest papers by meaning

• Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.90)
• Verifying Chain-of-Thought Reasoning via Its Computational Graph (ICLR 2026 · similarity 0.89)
• Rethinking Chain-of-Thought from the Perspective of Self-Training (ICML 2025 · similarity 0.89)
• Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning (ICML 2026 · similarity 0.89)
• Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning (AAAI 2026 · similarity 0.89)
• Reliable Chain-of-Thought via Prefix Consistency (NeurIPS 2026 · similarity 0.89)