Research map / Large language models / Reasoning and chain of thought
Uncovering the Gradient Geometry of Long CoT: A Spectral-guided Approach to Reasoning Distillation
Sinan Fan, Xiaofeng Sun, Chen Shen, Chenxi Huang, Shaotian Yan, Bing Wang, kaiyuan liu, Xiaosong Yuan, Liang Xie, Wenxiao Wang, Jun Zhang, Hongyang Chen and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a spectral-guided approach to long chain-of-thought reasoning distillation that treats gradient structure to separate essential reasoning tokens from noisy ones during SFT.
Problem
Standard SFT of student models on long CoT traces treats all tokens uniformly, including noisy, redundant, or instance-specific tokens, which leads to memorization of superficial patterns instead of generalizable reasoning.
Approach
Analyzes the gradient geometry of long-CoT distillation through a spectral lens and uses this spectral signal to guide training toward essential reasoning steps, rather than uniform token-level loss minimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use spectral-guided step selection instead of vanilla SFT | Qwen2.5-7B-Instruct · Math/reasoning average | accuracy (%): 40.6 -> 42.1 | Improved |
| use spectral-guided step selection instead of vanilla SFT | Qwen3-4B-Base · Math/reasoning average | accuracy (%): 48.6% to 51.3% (+2.7%) | Improved |
| use spectral-guided step selection instead of vanilla SFT | Qwen3-8B-Base · Math/reasoning average | accuracy (%): 54.0% to 56.7% (+2.7%) | Improved |
| use spectral-guided step selection instead of vanilla SFT | Qwen3-4B-Instruct-2507 · Math/reasoning average | accuracy (%): 55.7% to 61.2% (+5.5%) | Improved |
| add entropy-driven step prioritization (EDSP) instead of vanilla SFT | Qwen2.5-7B-Instruct · Math/reasoning average | accuracy (%): 40.60 -> 37.50 | Hurt |
| add random step masking at matching retention ratio (ablation) | Spectral-guided Learning (Qwen3-4B-Base) · Math/reasoning average | accuracy (%): 47.1 vs 48.6 (decreases performance by 1.5%) | Hurt |
| train only on low-spectral-strength residual steps (ablation) | Qwen3-4B-Base SFT · Math/reasoning average | accuracy (%): 44.9 vs 48.6 (3.6% degradation) | Hurt |
| use dynamic cumulative-energy truncation instead of static fixed-ratio selection (ablation) | Spectral-guided Learning (Qwen3-4B-Base) · Math/reasoning average | accuracy (%): 1.2% -> 2.4% average gain over Vanilla SFT | Improved |
| spectral-guided selection on AIME24 | Qwen3-4B-Instruct-2507 · AIME24 | accuracy (%): 51.7% to 59.2% | Improved |
| spectral-guided selection on AIME25 | Qwen3-8B-Base · AIME25 | accuracy (%): 29.2% to 35.0% | Improved |
| spectral-guided selection on MATH500 | Qwen3-4B-Instruct-2507 · MATH500 | accuracy (%): 83.9% to 88.4% (+4.5%) | Improved |
Experimental setup
Builds on: supervised fine-tuning chain-of-thought
Closest papers by meaning
- Efficient Reasoning via Constrained Optimization in Latent Space (NeurIPS 2026 · similarity 0.89)
- Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning (ICML 2026 · similarity 0.88)
- Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.88)
- Transformers Provably Solve Parity Efficiently with Chain of Thought (ICLR 2025 · similarity 0.87)
- TERMINATOR: Learning Optimal Exit Points for Early Stopping in Chain-of-Thought Reasoning (NeurIPS 2026 · similarity 0.87)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
