Research map / Large language models / Reasoning and chain of thought
CaTS: Calibrated Test-Time Scaling for Efficient LLM Reasoning
Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, Jiaxin Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaTS, a calibrated test-time scaling method for LLM reasoning that adaptively decides how many samples to draw per query instead of using a fixed N.
Problem
Best-of-N and Self-Consistency with majority voting use a fixed number of samples per query regardless of difficulty, wasting computation on simple questions and under-exploring complex ones.
Approach
Adaptive test-time scaling that calibrates how many responses to sample per query based on query complexity, allocating more computation to harder questions and less to easier ones. Specific mechanism not stated in the available abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| weight votes by self-calibrated confidence (CaTS-SC) | Self-Consistency on Llama-3.1-8B-Instruct · MathQA, budget 16 | accuracy (%): 73.7 -> 83.6 | Improved |
| early stop Best-of-N at a confidence threshold (CaTS-ES) | Best-of-N on DeepSeek-R1-Distill-1.5B · ARC Challenge, budget 16 | accuracy (%): 54.1 -> 66.5 | Improved |
| confidence-weighted adaptive SC (CaTS-ASC) | Adaptive Self-Consistency on Llama-3.1-8B-Instruct · MathQA, budget 16 | accuracy (%): 72.7 -> 81.9 | Improved |
| use CaTS-ES instead of Best-of-N | Qwen2.5-7B-Instruct · ARC Challenge, budget 16 | accuracy (%): 91.1 -> 90.8 | Hurt |
| use CaTS-ES with a small sample budget | Best-of-N on Qwen2.5-7B-Instruct · MathQA | accuracy | Hurt |
| distill self-consistency confidence into the model (Self-Calibration) | Llama-3.1-8B-Instruct · GSM8K | ECE: 13.70 -> 3.79 | Improved |
| Self-Calibration training | Qwen2.5-7B-Instruct · MathQA | accuracy (%): 49.85 -> 64.18 | Improved |
| use soft self-consistency score instead of P(True) or plain SC for labels (ablation) | Llama-3.1-8B-Instruct · GSM8K | ECE: 12.03 (P(True)), 4.48 (SC) -> 3.42 (SSC) | Improved |
| remove soft self-consistency from Self-Calibration (ablation) | Self-Calibration · Object Counting | ECE: 16.02 -> 9.69 (adding SSC) | Improved |
| remove entropy-based dynamic temperature (ablation) | Self-Calibration · Object Counting | accuracy (%): 62.88 -> 65.88 (adding EDT) | Improved |
| use SmoothL1 instead of MSE (ablation) | Self-Calibration · Object Counting | accuracy (%): 56.48 -> 65.88 | Improved |
| use self-calibrated confidence instead of a separate reward model for Best-of-16 | Llama-3.1-8B-Instruct · MathQA | accuracy (%): 82.1 -> 84.0 | No clear effect |
Experimental setup
Baselines: Best-of-N sampling Self-Consistency with majority voting
Builds on: Best-of-N sampling Self-Consistency
Closest papers by meaning
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.93)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.92)
- Representation Consistency for Accurate and Coherent LLM Answer Aggregation (NeurIPS 2025 · similarity 0.91)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.91)
- Inference Time Optimization with Confidence Dynamics (ICML 2026 · similarity 0.91)
- Consilience for Verifier-Free Test-Time Scaling (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
