Research map / Large language models / Reasoning and chain of thought
Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that cross-entropy fine-tuning hurts pass@N at large N through model overconfidence, and proposes Direct Coverage Optimization (DCO), a loss that directly optimizes pass@N coverage and limits confidence, improving MATH and MiniF2F/LeanDojo results.
Problem
Standard cross-entropy fine-tuning is misaligned with pass@N test-time search: pass@N accuracy can decrease with longer training, so training should be co-designed with the test-time compute strategy.
Approach
Derives a training loss -log(1-(1-p(y|x))^N) that directly maximizes the probability a correct answer appears in N samples; its gradient equals the CE gradient scaled by a factor that attenuates gradients on high-confidence examples, acting as an overconfidence regularizer. Extends it to step-wise DCO for theorem proving and approximate DCO for CoT via Monte Carlo estimates.
Key results
- On MATH with Llama-3-8B-base, CE pass@256 falls from 65.2% (epoch 1) to 44.5% (epoch 4) while pass@1 rises from 4.4% to 7.4%.
- On MiniF2F with pass@4k, ensembling DCO-step models with different N_eff reaches 43.6% vs 37.4% for CE loss.
- With CoT on MATH at epoch 5, DCO^a gives pass@64 of 64.3% vs 60.9% for CE and 35.4% for GRPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue CE training from epoch 1 to epoch 4 | Llama-3-8B-base · MATH test | pass@1: 4.4% -> 7.4% | Helped |
| Continue CE training from epoch 1 to epoch 4 | Llama-3-8B-base · MATH test | pass@256: 65.2% -> 44.5% | Hurt |
| Train with approximate DCO, N'=64, instead of CE | Llama-3-8B-base CoT fine-tuning · MATH CoT, epoch 5 | pass@1: 9.9$^{\pm0.1}$ -> 7.8$^{\pm0.1}$ | Hurt |
| Train with approximate DCO, N'=64, instead of CE | Llama-3-8B-base CoT fine-tuning · MATH CoT, epoch 5 | pass@64: 60.9$^{\pm0.6}$ -> \textbf{64.3}$^{\pm0.6}$ | Helped |
| Train with approximate DCO, N'=64, instead of GRPO | Llama-3-8B-base CoT fine-tuning · MATH CoT, epoch 5 | pass@64: 35.4$^{\pm0.3}$ -> \textbf{64.3}$^{\pm0.6}$ | Helped |
| Use step-wise DCO with N_eff=16 instead of CE (ablation) | Qwen2.5-Math-1.5B · Mathlib, pass@4k | proof success rate: 55.6% -> 56.5% | Helped |
| Use step-wise DCO with N_eff=8 instead of CE (ablation) | Qwen2.5-Math-1.5B · MiniF2F, pass@4k | proof success rate: 37.4% -> 39.5% | Helped |
| Ensemble models trained with different N_eff values | Qwen2.5-Math-1.5B · Mathlib, pass@4k | proof success rate: 55.6% -> 62.2% | Helped |
| Ensemble models trained with different N_eff values | Qwen2.5-Math-1.5B · MiniF2F, pass@4k | proof success rate: 37.4% -> 43.6% | Helped |
| Ensemble models trained with different N_eff values rather than CE at matched 5x test compute | Qwen2.5-Math-1.5B · Mathlib, pass@4k | proof success rate: 57.0% -> 62.2% | Helped |
| Ensemble models trained with different N_eff values rather than CE at matched 5x test compute | Qwen2.5-Math-1.5B · MiniF2F, pass@4k | proof success rate: 39.5% -> 43.6% | Helped |
| Apply naive DCO with N'=4k instead of CE | Qwen2.5-Math-1.5B · MiniF2F, pass@4k | proof success rate: 37.4% -> 38.7% | Helped |
Surprising findings
- Pass@N accuracy on MATH decreases with more CE training epochs even though pass@1 increases.
- GRPO-trained model shows strong overconfidence and degrades at large N, despite best pass@1.
Limitations
- Study limited to SFT with pass@N; other test-time strategies such as best-of-N not studied.
- Requires a verifier, restricting applicability to verifiable domains.
- Only mathematical reasoning evaluated; coding and other domains left for future work.
Experimental setup
Datasets: MATH LeanDojo
Benchmarks: MATH MiniF2F LeanDojo Mathlib
Metrics: pass@N coverage pass@1
Baselines: CE loss GRPO Llama-3-8B-base CE
Builds on: cross-entropy fine-tuning GRPO Dr. GRPO LeanDojo
Closest papers by meaning
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.92)
- SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling (AAAI 2026 · similarity 0.91)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.91)
- A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Consilience for Verifier-Free Test-Time Scaling (NeurIPS 2026 · similarity 0.90)
- Scaling Test-Time Compute Without Verification or RL is Suboptimal (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.