Research map / Large language models / Reasoning and chain of thought

Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies "Thought Leaps" (omitted intermediate steps) in math CoT data, builds ScaleQM+ to train CoT-Bridge (Qwen2.5-Math-7B) that detects leaps and fills missing steps, and shows bridged MetaMathQA/NuminaMath data improves SFT, distillation and RL starting points.

Problem

Expert-written mathematical chain-of-thought datasets often omit intermediate reasoning steps (thought leaps), which impairs LLM learning and generalization. The paper asks how to automatically detect such leaps and restore missing steps.

Approach

Formalizes the Thought Leap Bridge task (detect adjacent-step gaps and generate the missing steps), constructs ScaleQM+ by deleting intermediate steps from ScaleQuestMath to create supervised incomplete-complete pairs, trains CoT-Bridge to output leap positions and missing steps, and applies it as a plug-and-play augmentation for existing CoT datasets.

Key results

• CoT-Bridge SFT on NuminaMath improves Meta-Llama3.1-8B average accuracy by +5.87% (43.87 to 49.74) over direct SFT.
• On the leap identification task, CoT-Bridge reaches 78.02% precision and 1.61% redundancy, versus 33.99% precision and 33.73% redundancy for Qwen2.5-Instruct-72B.
• Bridged distilled data raises Qwen2.5-Math-1.5B-based average from 55.23% to 58.25% (+3.02%); RL from a NuminaMath-Bridge start reaches 63.98% vs 60.88% for NuminaMath.

Measured effects reported in the paper

Surprising findings

• Zero-shot bridging with the 72B instruct model sometimes degrades performance, e.g. AMC23 drops by 15 points under Llama + MetaMathQA.
• Random-position bridging yields highly variable results and can hurt performance on some benchmarks, showing accurate leap localization matters.

Limitations

• Bridged steps are not guaranteed correct; noise is introduced in automated data construction.
• Experiments are limited to small-to-medium models; 32B and 72B models were not evaluated.
• CoT-Bridge is trained only on mathematical data, with OOD evaluation limited to logical reasoning tasks close to math.

Experimental setup

Datasets: ScaleQuestMath MetaMathQA NuminaMath-CoT GSM8K MATH DAPO-Math-17K

Benchmarks: GSM8K MATH500 GaoKao2023EN MathOdyssey OlympiadBenchEN AMC23 FOLIO LogicQA ProofWriter ReClor RuleTaker

Metrics: accuracy leap precision leap recall redundancy rate position-aware BERTScore PRM score invalid response rate

Baselines: Direct SFT Qwen2.5-Instruct-7B zero-shot bridging (QwenBridger-S) Qwen2.5-Instruct-72B zero-shot bridging (QwenBridger-L) CoT-Bridge-Random Qwen2.5-Math-Instruct-1.5B Oat-Zero-1.5B

Builds on: ScaleQuestMath MetaMathQA NuminaMath-CoT GRPO LlamaFactory

Closest papers by meaning

• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.92)
• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.92)
• Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns (ICLR 2026 · similarity 0.92)
• Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.91)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.91)