Research map / Large language models / Reasoning and chain of thought

Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically and empirically that under outcome-only supervision, scaling training-time reasoning length (RL token budget or looped Transformer loops) can keep improving OOD accuracy after ID accuracy saturates.

Problem

Under outcome-only supervision, how training-time reasoning length (CoT token budget in RL or loop count in looped Transformers) affects which solution is learned and its OOD generalization, when ID validation performance has already plateaued.

Approach

Views increasing training-time reasoning length as increasing self-iteration, which changes the induced hypothesis class; proves two mechanisms (inductive bias from k-fold self-composition, and regularization suppressing persistent shortcut components) under which OOD loss decreases with k while ID loss stays flat.

Key results

• In looped Transformers on 4-hop problems, ID accuracy saturates near 100% by about 2 loops while OOD accuracy keeps improving up to about 44 loops.
• RL fine-tuning Qwen2.5-1.5B-Instruct: ID accuracy peaks at a 256-token budget and slightly drops thereafter, while OOD accuracy steadily improves with larger budgets (Pass@10).
• Theory: for the stylized example, increasing k from 1 to 8 provably reduces OOD loss by at least 0.754 with almost no ID change.

Surprising findings

• Small RL token budgets (128/256) keep reward near zero for long stretches before a sudden drop in response length, after which the model learns to answer with little step-by-step reasoning, achieving comparable ID accuracy with worse OOD accuracy.
• Looped Transformers keep gaining OOD accuracy at very large loop counts (up to about 44) after ID saturation at about 2 loops.

Limitations

• Theory relies on stylized toy settings and assumptions (e.g., structured/shortcut decomposition, regularization with the right bias) not verified for full LLMs.
• The authors acknowledge that persistent seemingly natural shortcuts not penalized by the regularizer may defeat the effect.
• RL experiments use a single 1.5B model and a small set of math datasets with OOD defined by problem topic.

Experimental setup

Datasets: Polynomial Roots (adapted variant) Mixed Operation p-hop induction synthetic task

Benchmarks: Polynomial Roots (ID) Mixed Operation (OOD) p-hop induction

Metrics: accuracy pass@10 pass@1

Builds on: GRPO looped transformers in-context learning of linear regression via linear transformers

Closest papers by meaning

• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.90)
• Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.90)
• Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning (NeurIPS 2025 · similarity 0.90)
• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.90)
• Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning (NeurIPS 2025 · similarity 0.90)