Research map / Large language models / Reasoning and chain of thought

Conformal Thinking: Risk Control for Reasoning on a Compute Budget

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Conformal Thinking, which uses distribution-free risk control (UCB) to calibrate an upper confidence threshold and a parametric dynamic lower threshold for early stopping of LLM reasoning under a user-specified risk target.

Problem

Setting token budgets and stopping thresholds for adaptive reasoning in LLMs is hard because thresholds are uninterpretable; early termination risks errors, and existing methods only stop when confident, wasting tokens on unsolvable instances.

Approach

Reframe budget setting as risk control: an upper confidence threshold stops when confident, and a novel parametric sigmoid lower threshold stops when confidence is not rising fast enough. Thresholds are selected with UCB (Hoeffding finite-sample correction) to meet a user risk tolerance while minimizing an efficiency loss.

Key results

• UCB calibration keeps realized test risk below the user target across 40 random validation-test splits, whereas naive cross-validation frequently exceeds the target, especially for false negative risk.
• On mixed solvable:unsolvable sets, adding the lower threshold (Lower+Upper) shifts the accuracy-token curve left, giving similar accuracy with fewer tokens than Upper-only when unsolvable instances are common.
• Ensembling signals (confidence, EAT, probe, token count) by selecting the most efficient feasible signal per risk target improves the accuracy-token trade-off over single signals.

Surprising findings

• Upper-only early stopping clusters near the maximum token budget when unsolvable instances are common because many runs never reach the confidence cutoff.
• Length shift from short validation to long test traces particularly breaks the false negative (lower threshold) risk guarantee.

Limitations

• Guarantees rely on monotonicity of risks in hyperparameters and on the upper risk being unaffected by the lower threshold, which are only verifiable in large-sample regimes.
• Evaluation limited to math and science reasoning because uncertainty signals do not transfer to less structured reasoning and outputs.
• Distribution shift (length or dataset) can break the lower-threshold risk control.

Experimental setup

Datasets: AIME 1983-2025 DeepScaleR subset GPQA-Diamond MathVision

Benchmarks: AIME DeepScaleR GPQA-Diamond MathVision

Metrics: false positive risk false negative risk accuracy vs total tokens empirical test risk vs target risk

Baselines: Naive cross-validation thresholding Upper-only threshold Lower-only threshold Uniform token budget Thought Calibration PAC Reasoning

Builds on: learn-then-test upper confidence bound risk control early-exit neural networks EAT entropy after think

Closest papers by meaning

• Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models (ICLR 2026 · similarity 0.89)
• Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge (ICML 2026 · similarity 0.88)
• The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs (ICML 2026 · similarity 0.88)
• Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning (NeurIPS 2025 · similarity 0.88)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• Optimizing Anytime Reasoning via Budget Relative Policy Optimization (NeurIPS 2025 · similarity 0.88)