Research map / Large language models / Reasoning and chain of thought
Chain of Execution Supervision Promotes General Reasoning in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds TracePile, a 2.6M-sample corpus of natural-language step-by-step code-execution traces (Chain of Execution) and shows it improves math, code, logic and algorithm reasoning when used for continued pretraining or two-stage finetuning of 7-8B models.
Problem
Raw code is a rich source of reasoning patterns, but its reasoning signals are implicit and entangled with syntax, so training directly on raw code is suboptimal for general LLM reasoning.
Approach
Convert code execution into explicit step-by-step CoT-style narrations (Chain of Execution), augmented with variable-tracing queries and LLM-based code rewrites, and use this as an intermediate training stage before or alongside general instruction tuning.
Key results
- Continued pretraining raises LLaMA-3-8B average over nine math benchmarks from 54.8 to 63.4.
- Two-stage finetuning raises LLaMA-3-8B math average from 55.3 to 61.7 and Qwen-2.5-7B from 73.6 to 75.4.
- Under continued pretraining, GraphWiz and GraphInstruct gains exceed 40 points for several base models (e.g., Qwen-2.5-7B GraphWiz 36.6 to 50.5, Llama-3-8B GraphWiz 4.5 to 46.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| continue-pretrain on TracePile | LLaMA-3-8B · Math average | accuracy: 54.8 -> 63.4 | Helped |
| continue-pretrain on TracePile | LLaMA-3.1-8B · Math average | accuracy: 57.4 -> 64.8 | Helped |
| fine-tune on TracePile, then TuluSFT | Qwen-2.5-Coder-7B · LiveCodeBench | accuracy: 27.2 -> 29.9 | Helped |
| use TracePile CoE data instead of alternative supervision | Qwen-2.5-Coder-7B · LiveCodeBench-Output | accuracy: 29.8 -> 35.5 | Helped |
| add mathematical data (ablation) | TracePile · Mathematical average | accuracy: 60.6 -> 62.4 | Helped |
| add mathematical data (ablation) | TracePile · Code average | accuracy: 26.2 -> 30.6 | Helped |
| add classical algorithm data (ablation) | TracePile · Code average | accuracy: 23.1 -> 30.6 | Helped |
| add algorithmic competition data (ablation) | TracePile · Code average | accuracy: 24.9 -> 30.6 | Helped |
| add algorithmic competition data (ablation) | TracePile · Algorithm average | accuracy: 35.0 -> 40.3 | Helped |
| add query diversification (ablation) | TracePile · Code average | accuracy: 30.0 -> 30.6 | Helped |
| add query diversification (ablation) | TracePile · Algorithm average | accuracy: 37.9 -> 40.3 | Helped |
| add code diversification (ablation) | TracePile · Code average | accuracy: 28.2 -> 30.6 | Helped |
Surprising findings
- Rejection-sampled TracePile++ improves math but plateaus or regresses on code, logic and algorithmic reasoning.
- Some base models without instruction tuning cannot follow complex instructions (e.g., CLRS), so they could not be evaluated after continued pretraining only.
Limitations
- Relies on large instruction-tuned models (Qwen-2.5-72B) for generating traces, adding cost and possible bias.
- 8k token cutoff may discard long, informative traces.
- Coverage skews toward math and algorithmic reasoning; no commonsense or world-knowledge domains.
Experimental setup
Datasets: Codeforces Python submissions CLRS-text generator GraphInstruct generator OpenMath Tulu3-SFT
Benchmarks: GSM8K MATH GSM-Hard SVAMP ASDIV MAWPS MinMATH MMLU-STEM TABMWP MATHQA SAT LiveCodeBench
Metrics: accuracy
Baselines: OpenMathInstruct-1 OpenMathInstruct-2 WebInstruct CodeI/O CodeI/O++
Builds on: CodeI/O OpenMath Tulu 3 SFT CLRS
Closest papers by meaning
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.91)
- Chain of Code: Reasoning with a Language Model-Augmented Code Emulator (ICML 2024 · similarity 0.90)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.90)
- CodeIO: Condensing Reasoning Patterns via Code Input-Output Prediction (ICML 2025 · similarity 0.89)
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.89)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.