Research map / Large language models / Reasoning and chain of thought
Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes a 50K code-generation SFT dataset with separate CoT and code, and finds that training models to output code first and then a CoT explanation ("code precede CoT") beats the traditional reasoning-first order.
Problem
Whether classic chain-of-thought ordering (reasoning before code) helps supervised fine-tuning of LLMs for code generation, and how the position of CoT relative to code affects learning.
Approach
Builds a dataset where NL problem, CoT reasoning, and code are separated and compares four SFT strategies, finding that placing code before the CoT explanation yields better code generation than CoT-first, treating CoT as an explanation rather than a reasoning process.
Key results
- Code-precede-CoT reaches 71.88 average on EvalPlus vs 65.43 for code-follow-CoT (about 9.86% relative improvement) with DeepSeek-Coder-6.7B-Base.
- On LeetCode Contest benchmark (180 problems), Cprecede gets 25.0% overall vs 16.1% for Cfollow and 19.4% for code without CoT.
- Cprecede wins 70.3% of model-level win rates in EvalPerf and gets the highest MultiPL-E average (50.11% vs 47.29% for Cfollow).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Put code before CoT instead of putting CoT before code (ablation) | SFT of DeepSeek-Coder-Base-6.7B · EvalPlus average | Pass@1: 65.43 -> 71.88 | Helped |
| Train on code without CoT rather than the Seed dataset (ablation) | SFT of DeepSeek-Coder-Base-6.7B · EvalPlus average | Pass@1: 68.40 -> 70.88 | Helped |
| Put CoT before code rather than using the Seed dataset (ablation) | SFT of DeepSeek-Coder-Base-6.7B · EvalPlus average | Pass@1: 68.40 -> 65.43 | Hurt |
| Put code before CoT rather than using the Seed dataset (ablation) | SFT of DeepSeek-Coder-Base-6.7B · EvalPlus average | Pass@1: 68.40 -> 71.88 | Helped |
| Put code before CoT rather than putting CoT before code (ablation) | SFT of DeepSeek-Coder-Base-6.7B · LiveCodeBench | Pass@1: 20.69 -> 21.08; 22.31 -> 22.56; 23.07 -> 24.16 | Helped |
| Put code before CoT rather than putting CoT before code (ablation) | SFT of DeepSeek-Coder-Base-6.7B · BigCodeBench | Pass@1: 36.05 -> 38.33; 15.54 -> 16.22 | Helped |
| Use a GPT-4o-0806-synthesized dataset (ablation) | SFT strategies · EvalPlus average | Pass@1: 65.95 -> 71.33 | Helped |
| Synthesize training data in an alternate order without reference code (ablation) | SFT strategies · EvalPlus average | Pass@1: 61.74 -> 67.35 | Helped |
| Synthesize training data from Stack code (ablation) | SFT strategies · EvalPlus average | Pass@1: 63.13 -> 68.65 | Helped |
| Remove function or class signatures from the code (ablation) | SFT training data · EvalPlus average | Pass@1 | Hurt |
| Mix Cfollow and Cprecede training patterns (ablation) | SFT of the base model · EvalPlus average | Pass@1 | Hurt |
| Replace code variable names with random, meaningless strings (ablation) | SFT training data · EvalPlus average | Pass@1 | Hurt |
Surprising findings
- Traditional CoT-before-code SFT (Cfollow) underperforms even training on code without any CoT, on EvalPlus and LiveCodeBench.
- Mixing both orderings in training degrades performance relative to either single strategy.
Limitations
- Synthetic CoT data generated by a single teacher model (DeepSeek-V2.5) with reasoning tied to reference code, possibly biasing results; partially mitigated with a second teacher and a no-code-hint variant.
- Evaluation focuses on SFT with ~51K samples; generalization to pretraining or RL-trained reasoning models is not tested.
Experimental setup
Datasets: Magicoder-OSS-Instruct ShareGPT Python subset Evol-CodeAlpaca Evol-Instruct-Code CodeExercise-Python Codefuse-Evol-Instruct Stack
Benchmarks: HumanEval HumanEval+ MBPP MBPP+ LiveCodeBench BigCodeBench EvalPerf MultiPL-E LeetCode Contest
Metrics: Pass@1 Pass@k DPS CodeBLEU BLEU win rate
Baselines: Seed dataset SFT Code without CoT Code follow CoT
Builds on: chain-of-thought prompting context distillation STaR SelfCodeAlign
Closest papers by meaning
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.92)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.92)
- Understanding Chain-of-Thought in LLMs through Information Theory (ICML 2025 · similarity 0.90)
- What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.90)
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.90)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.