AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Uncovering the Gradient Geometry of Long CoT: A Spectral-guided Approach to Reasoning Distillation

Sinan Fan, Xiaofeng Sun, Chen Shen, Chenxi Huang, Shaotian Yan, Bing Wang, kaiyuan liu, Xiaosong Yuan, Liang Xie, Wenxiao Wang, Jun Zhang, Hongyang Chen and 1 more

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a spectral-guided approach to long chain-of-thought reasoning distillation that treats gradient structure to separate essential reasoning tokens from noisy ones during SFT.

Problem

Standard SFT of student models on long CoT traces treats all tokens uniformly, including noisy, redundant, or instance-specific tokens, which leads to memorization of superficial patterns instead of generalizable reasoning.

Approach

Analyzes the gradient geometry of long-CoT distillation through a spectral lens and uses this spectral signal to guide training toward essential reasoning steps, rather than uniform token-level loss minimization.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use spectral-guided step selection instead of vanilla SFTQwen2.5-7B-Instruct · Math/reasoning averageaccuracy (%): 40.6 -> 42.1Improved
use spectral-guided step selection instead of vanilla SFTQwen3-4B-Base · Math/reasoning averageaccuracy (%): 48.6% to 51.3% (+2.7%)Improved
use spectral-guided step selection instead of vanilla SFTQwen3-8B-Base · Math/reasoning averageaccuracy (%): 54.0% to 56.7% (+2.7%)Improved
use spectral-guided step selection instead of vanilla SFTQwen3-4B-Instruct-2507 · Math/reasoning averageaccuracy (%): 55.7% to 61.2% (+5.5%)Improved
add entropy-driven step prioritization (EDSP) instead of vanilla SFTQwen2.5-7B-Instruct · Math/reasoning averageaccuracy (%): 40.60 -> 37.50Hurt
add random step masking at matching retention ratio (ablation)Spectral-guided Learning (Qwen3-4B-Base) · Math/reasoning averageaccuracy (%): 47.1 vs 48.6 (decreases performance by 1.5%)Hurt
train only on low-spectral-strength residual steps (ablation)Qwen3-4B-Base SFT · Math/reasoning averageaccuracy (%): 44.9 vs 48.6 (3.6% degradation)Hurt
use dynamic cumulative-energy truncation instead of static fixed-ratio selection (ablation)Spectral-guided Learning (Qwen3-4B-Base) · Math/reasoning averageaccuracy (%): 1.2% -> 2.4% average gain over Vanilla SFTImproved
spectral-guided selection on AIME24Qwen3-4B-Instruct-2507 · AIME24accuracy (%): 51.7% to 59.2%Improved
spectral-guided selection on AIME25Qwen3-8B-Base · AIME25accuracy (%): 29.2% to 35.0%Improved
spectral-guided selection on MATH500Qwen3-4B-Instruct-2507 · MATH500accuracy (%): 83.9% to 88.4% (+4.5%)Improved

Experimental setup

Builds on: supervised fine-tuning chain-of-thought

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy