Research map / Large language models / Instruction and fine-tuning

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Fine-tuning for reasoning data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that extending a model's long-context capacity (via RoPE theta scaling or model merging with a 1M-context model) before reasoning SFT improves math and cross-domain reasoning accuracy, and proposes a long-context-first recipe for reasoning fine-tuning.

Problem

Whether a model's long-context capacity limits its reasoning ability, and whether improving long-context ability before reasoning SFT improves downstream reasoning performance.

Approach

Treats long-context capacity as a controllable variable: varies RoPE theta scaling or merges a base model with a 1M-context model to obtain models with identical SFT data but different long-context strength, then measures post-SFT reasoning accuracy and proposes extending context before reasoning SFT.

Key results

• LLaMA3-8B-Instruct with RoPE theta x16 reaches 59.36 average MATH500 accuracy after SFT versus 54.80 for x1 (32k NIAH 77.05 vs 0).
• Qwen2.5-Math-7B-Instruct recipe (RoPE x16 merged 0.3 with 1M model) raises MATH500 average from 85.04 to 88.70 and AIME22-24 from 15.00 to 28.00.
• Phi-4 with RoPE x16 improves MATH500 after SFT from 88.62 (x1) to 89.90.

Measured effects reported in the paper

Surprising findings

• Stronger long-context models produce shorter average outputs after SFT, attributed to a larger share of correct (shorter) answers.
• Pure 1M-context merge (ratio 1.0) shows degraded reasoning relative to moderate merge ratios.

Limitations

• Analysis limited to 7B-8B models (and one 14B model) and does not cover larger scales.
• Focuses on supervised fine-tuning; interaction with reinforcement learning left open.
• Correlational evidence; controlled variation of long-context capacity via RoPE and merging may affect other properties.

Experimental setup

Datasets: OpenR1-Math-220K OpenThoughts3-1.2M NuminaMath-CoT DeepMath-103K

Benchmarks: MATH500 AIME22-24 GSM8K GPQA LiveCodeBench MMLU-STEM Needle-in-a-Haystack LongBench RULER

Metrics: pass@1(5) accuracy needle-in-a-haystack accuracy average output length

Baselines: RoPE theta x1 DeepSeek-R1-Distill-Qwen-7B OpenR1-Qwen-7B OpenThinker-7B

Builds on: YaRN NTK scaling Position Interpolation

Closest papers by meaning

• InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models (ICLR 2026 · similarity 0.91)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
• Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities (AAAI 2026 · similarity 0.90)
• Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision (NeurIPS 2026 · similarity 0.90)
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.90)
• Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens (NeurIPS 2025 · similarity 0.90)