Research map / Large language models / Reasoning and chain of thought
Demystifying Long Chain-of-Thought Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies how long chain-of-thought reasoning emerges in LLMs via SFT and PPO-based RL, proposing a cosine length-scaling reward with repetition penalty and showing noisy web-extracted verifiable data helps OOD reasoning.
Problem
Long chain-of-thought (CoT) reasoning that enables inference-time scaling is emerging via RL, but the conditions under which long CoTs arise and how to train RL stably for them remain unclear.
Approach
Uses a cosine length-scaling reward (correct/wrong rewards interpolated by CoT length) with an N-gram repetition penalty to stabilize RL-driven CoT growth, and studies SFT initialization from distilled long CoT data plus filtered noisy web-extracted (WebInstruct) verifiable data for RL.
Key results
- Long CoT SFT on Llama-3.1-8B scales past 70% on MATH-500 and keeps improving at 3.5B tokens, while short CoT SFT saturates below 55%.
- RL from long CoT SFT with emergent-pattern distillation improves Llama-3.1-8B MATH-500 from 54.1 to 59.4 and MMLU-Pro-1k from 32.0 to 34.6.
- Filtered WebInstruct with rule-based verifier RL reaches 48.6 on MATH-500 and 41.4 on MMLU-Pro-1k, and gains up to 2.9% on TheoremQA and 6.8% on MMLU-Pro-1k over MATH-only training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use emergent rather than constructed long-CoT SFT patterns, then apply RL (ablation) | Llama-3.1-8B · MMLU-Pro-1k | accuracy: 19.2 -> 34.6 | Helped |
| Mix 50% WebInstruct into MATH long-CoT SFT data (ablation) | Llama-3.1-8B · MMLU-Pro-1k | accuracy: Adding WebInstruct data to long CoT SFT yields a substantial 5–10% absolute accuracy gain on MMLU-Pro-1k | Helped |
| Use a 50% MATH and 50% WebInstruct mixture rather than MATH-only SFT (ablation) | Llama-3.1-8B · average across MATH-500, AIME 2024, TheoremQA, and MMLU-Pro-1k | accuracy: 30.8 -> 32.1 | Helped |
| Filter noisy prompts for short-form answers before using a rule-based verifier (ablation) | Llama-3.1-8B RL with WebInstruct · TheoremQA | accuracy: 25.9 -> 28.1 | Helped |
| Use a rule-based verifier on unfiltered noisy prompts instead of a model-based verifier (ablation) | Llama-3.1-8B RL with WebInstruct · MMLU-Pro-1k | accuracy: 40.4 -> 35.1 | Hurt |
| Apply RL after long-CoT SFT instead of direct RL from the base model | Qwen2.5-Math-7B · MMLU-Pro-1k | accuracy: 19.7 -> 40.6 | Helped |
| Use long-CoT SFT rather than direct RL from the base model | Qwen2.5-Math-7B · MATH-500 | accuracy: 77.4 -> 85.9 | Helped |
| Use the cosine length-scaling reward with repetition penalty rather than classic reward (ablation) | Llama-3.1-8B with long-CoT SFT · downstream tasks | accuracy | Helped |
| Add an N-gram repetition penalty to cosine-reward RL (ablation) | Llama-3.1-8B with long-CoT SFT · downstream tasks | accuracy | Helped |
| Use an 8K rather than 4K context window (ablation) | Llama-3.1-8B with long-CoT SFT and RL · downstream tasks | accuracy | Helped |
| Increase the context window from 8K to 16K with the same training samples (ablation) | Llama-3.1-8B with long-CoT SFT and RL · downstream tasks | accuracy | Hurt |
| Lower the correctness-reward discount factor to 0.99 (ablation) | Llama-3.1-8B with cosine reward and repetition penalty · MATH-500 | accuracy: 58.0 -> 47.9 | Hurt |
Surprising findings
- RL from Qwen2.5-Math-7B base does not increase reflection keywords like "recheck" or "alternatively" despite rising accuracy, and length growth may be driven by KL penalty reversion rather than new long CoT ability.
- Training with more compute can lead to reward hacking via repetition, and a larger context window (16K) underperformed 8K under equal training samples.
Limitations
- Experiments are restricted to 7-8B models; scaling to 32B was infeasible due to GPU requirements.
- Evaluation is largely math-focused with a small AIME 2024 set, giving high variance.
- Several RL results from base model are sensitive to inference temperature and PPO/REINFORCE++ tuning.
Experimental setup
Datasets: MATH WebInstruct OpenWebMath
Benchmarks: MATH-500 AIME 2024 TheoremQA MMLU-Pro-1k
Metrics: accuracy
Baselines: Classic Reward Short CoT SFT RL from base model (Qwen2.5-Math-7B) Constructed long CoT (Action Prompting)
Builds on: PPO REINFORCE++ OpenRLHF DeepSeek-R1
Closest papers by meaning
- When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.95)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.93)
- Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.92)
- Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.92)
- Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning (NeurIPS 2025 · similarity 0.92)
- Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.