Research map / Large language models / Reasoning and chain of thought

RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how RL and SFT reshape LLM reasoning paths at trajectory level (unique correct/incorrect clusters) and step level (sentence-cluster reasoning graphs with exponential decay rates and topology metrics) for 1.5B-14B distilled and RL-tuned models.

Problem

Accuracy-based evaluations (e.g., Pass@k) do not reveal how RLVR and SFT change the underlying reasoning process of LLMs, and it is unclear why SFT-then-RL works.

Approach

Measures reasoning paths at two granularities: clusters unique trajectories with chrF/UPGMA to show RL compresses incorrect paths while SFT expands correct ones, and builds sentence-level reasoning graphs whose exponential decay rates show RL concentrates functionality into fewer steps while SFT spreads it out.

Key results

• Trajectory-level clustering shows RL reduces unique incorrect trajectories while SFT increases unique correct trajectories, with SFT+RL combining both effects.
• Exponential decay rate beta for visitation frequency, degree, and betweenness rises under RL (about 2.5x) and falls under SFT (to about one-third).
• RL-trained graphs show high edge density, low clustering, and high Freeman centralization; SFT and SFT+RL graphs show low modularity, high global efficiency and algebraic connectivity.

Surprising findings

• SFT alone preserves incorrect trajectories, so SFT does not guarantee Pass@1 gains.
• RL also reduces the number of correct trajectories, explaining why Base Pass@k overtakes RL at large k.

Limitations

• Training datasets are not controlled across RL and SFT models, so differences partly reflect data differences.
• Graphs are dense because sentence clustering produces high edge density; sparsification was needed as a check.
• Local graphlet structure alone does not explain the performance gaps between RL and SFT models.

Experimental setup

Datasets: AIME24 AIME25 AMC23 HumanEval s1k-1.1

Benchmarks: AIME24 AIME25 AMC23 HumanEval

Metrics: Pass@k Pass@1 number of unique correct trajectories number of unique incorrect trajectories exponential decay rate global efficiency modularity algebraic connectivity graphlet proportion

Baselines: Base model (Qwen2.5-Math) DeepSeek-R1-Distill-Qwen AceReason-Nemotron SimpleRL-Zoo Oat-Zero

Builds on: GRPO Dr.GRPO Pass@k chain-of-thought

Closest papers by meaning

• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.94)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.94)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.94)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.94)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.93)
• RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning (ICLR 2026 · similarity 0.93)