Research map / Large language models / Reasoning and chain of thought

On the Overscaling Curse of Parallel Thinking: System Efficacy Contradicts Sample Efficiency

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies an "overscaling curse" in parallel thinking where a global sampling budget exceeds many samples' optimal budgets, and proposes LanBo, a layer-wise latent probe that predicts per-sample budgets before decoding (Pre-decoding Budget Adaptation) to improve budget utilization and cut latency/memory.

Problem

Parallel thinking evaluations use one global sample budget chosen to maximize dataset accuracy, which leaves most samples with low budget utilization; existing adaptive budgeting methods add blocking overhead during decoding.

Approach

Formalize the gap between system-optimal and sample-optimal budgets, then train lightweight MLP probes on intermediate last-token hidden states to predict each sample's optimal budget before decoding, combining layers by inverse-error weighting so parallel decoding stays fully batched.

Key results

• LanBo raises budget utilization by roughly 3-6x on average while maintaining dataset accuracy, with utilization hovering near 1 after prediction.
• On Qwen3-4B, LanBo reduces latency to about 0.45-0.79x of standard parallel decoding and memory to roughly 0.10-0.57x, with accuracy within about 2% of STD.
• In 79 of 84 system-aggregation groups, expected budget utilization is below 1, indicating the overscaling curse is prevalent.

Surprising findings

• Severe overscaling is driven by a small Type-4 (monotonically increasing) sample subset whose large optimal budgets pull the system-optimal global budget upward.
• Existing decoding-time adaptive methods (InAda) can be slower than standard parallel decoding because serializing parallel requests hurts latency more than budget reduction helps.

Limitations

• Evaluated only on four models and seven datasets, with the predictor trained per model.
• Relies on a diagonal approximation to the error covariance for layer weighting, with cross-layer correlations ignored.
• Theorem relies on assumed transfer of dominance conditions and on negligible Type-5 contribution.

Experimental setup

Datasets: DeepMath-103K WebInstruct-verified

Benchmarks: MATH500 AMC AIME24 AIME25 GPQA MMLU-Pro BrowseComp

Metrics: expected budget utilization dataset accuracy memory ratio vs STD latency ratio vs STD predictor MAE

Baselines: STD. (standard parallel decoding) Adaptive-Consistency (AC) ESC DSC DeepConf

Builds on: self-consistency adaptive-consistency DeepConf

Closest papers by meaning

• Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.87)
• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models (ICML 2026 · similarity 0.87)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.87)
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.86)
• D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting (NeurIPS 2026 · similarity 0.86)
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning (NeurIPS 2025 · similarity 0.86)