Research map / Large language models / Reasoning and chain of thought
Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Caco fine-tunes a code model (CodeGen) on executable code chain-of-thoughts, samples and verifies new code CoTs, then back-translates them into verified natural-language problems and CoTs to build a 1.3M-sample math instruction dataset.
Problem
Existing chain-of-thought data for LLM reasoning relies on natural-language traces that are hard to verify and expensive to scale, while prior code-assisted methods generalize poorly beyond predefined math problems.
Approach
Unify seed solutions as executable Python code templates, fine-tune an unconditional code generator on them to sample diverse new code CoTs, filter by execution, and reverse-translate the verified code into natural-language problems and language CoTs checked by answer-matching and CoT-consistency verification.
Key results
- Caco-1.3M on Qwen2.5-Math-7B reaches 67.7 average across six math benchmarks, roughly matching Qwen2.5-Math-7B-Instruct (63.6) and above the best listed baseline (NuminaMath 59.3).
- Caco-1.3M on LLaMA3-8B averages 57.3 versus 39.7 for DartMath, as reported in the paper.
- In a matched 300K control, Caco-300K beats a Qwen72B-distilled natural-language CoT baseline (66.2 vs 65.5 AVG for Qwen-7B) and a STaR-style self-improvement baseline (53.2 vs 48.9 for DS-7B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase generated training data from 596K to 1.3M (ablation) | Caco on Qwen2.5-Math-7B · Average across math benchmarks | accuracy: 66.6 -> 67.7 | Helped |
| increase generated training data from 109K to 1.3M (ablation) | Caco on LLaMA3-8B · Average across math benchmarks | accuracy: 46.7 -> 57.3 | Helped |
| add verification filtering to the training data pipeline (ablation) | Caco-trained LLaMA · Math benchmarks | average accuracy: 20.8 -> 21.8 | Helped |
| add verification filtering to the training data pipeline (ablation) | Caco-trained LLaMA · OlympiadBench-Math | accuracy: 18.7, outperforming its non-verified counterpart by 1.1 points | Helped |
| use Caco-generated data instead of teacher-distilled natural-language CoTs at 300K samples | Qwen-7B · Average across math benchmarks | accuracy: 65.5 -> 66.2 | Helped |
| scale Caco data from 300K to 1.3M samples (ablation) | Qwen-7B · Average across math benchmarks | accuracy: 66.2 -> 67.7 | Helped |
| use Caco-generated data instead of seed self-improvement at 300K samples | DS-7B · Average across math benchmarks | accuracy: 48.9 -> 53.2 | Helped |
| use Caco-generated data instead of seed self-improvement at 300K samples | Qwen-7B · Average across math benchmarks | accuracy: 52.6 -> 66.2 | Helped |
| train on Caco-1.3M instead of DartMath | LLaMA3-8B · Average across math benchmarks | accuracy: 39.7 -> 57.3 | Helped |
| train on Caco-1.3M instead of Qwen2.5-Math-7B-Instruct | Qwen2.5-Math-7B · Average across math benchmarks | accuracy: 63.6 -> 67.7 | Helped |
Surprising findings
- Jointly generating the instruction and language CoT from code led to low-quality solutions, which motivated the two-stage generation design.
- Caco-1.3M on Qwen2.5-Math-7B gave only a modest gain over the 596K version (67.7 vs 66.6 AVG), suggesting diminishing returns on already strong base models.
Limitations
- Performance limited by predefined problem types used in CodeGen training; struggles with innovative or unconventional problems.
- Generated code is used only for filtering, not directly as training data; back-translation can lose detail.
- Scope focused on mathematical and algorithmic reasoning, with only a small science extension.
Experimental setup
Datasets: MATH DeepScaleR BigMath KodCode MegaScience GSM8K CollegeMath DeepMind-Mathematics OlympiadBench TheoremQA
Benchmarks: MATH GSM8K CollegeMath DeepMind-Mathematics OlympiadBench-Math TheoremQA AIME24 HumanEval+ ARC-c BBH KorBench AGIEval
Metrics: Pass@1 accuracy average accuracy solvability ratio correctness ratio
Baselines: MetaMath MMIQC NuminaMath MathFusion RFT DART-Math LLaMA3-8B-Instruct Qwen2.5-Math-7B-Instruct DeepSeekMath-7B-RL
Builds on: DartMath MathFusion Program of Thoughts STaR
Closest papers by meaning
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.92)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.92)
- Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning (NeurIPS 2025 · similarity 0.92)
- Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.92)
- CodeIO: Condensing Reasoning Patterns via Code Input-Output Prediction (ICML 2025 · similarity 0.92)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.