Research map / Large language models / Reasoning and chain of thought

CodeIO: Condensing Reasoning Patterns via Code Input-Output Prediction

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CodeI/O, which trains LLMs to predict code inputs/outputs with natural-language CoTs (distilled from DeepSeek-V2.5) as a first-stage intermediate training step, plus CodeI/O++ with execution-verified multi-turn revision, improving general reasoning across 14 benchmarks.

Problem

Reasoning training data beyond math and code generation is sparse and fragmented, and raw code training carries reasoning signals that are implicit and noisy.

Approach

Transform raw code into executable functions with queries, generated inputs and execution-verified outputs, then train models to predict outputs from inputs (or feasible inputs from outputs) entirely in natural-language CoT, decoupling reasoning from code syntax; a two-stage training with a separate instruction-tuning stage follows, and execution feedback drives a second revision turn (CodeI/O++).

Key results

• CodeI/O improves Qwen 2.5 Coder 7B average from 54.8 (2nd stage only) to 57.2, and CodeI/O++ reaches 57.7 on the same model.
• On LLaMA 3.1 8B, CodeI/O++ raises average from 49.3 to 52.1; on Gemma 2 27B, CodeI/O++ raises average from 59.5 to 61.5.
• Training on raw PyEdu (7.7M) yields only minor or negative gains (e.g., 54.8 to 54.8 on Qwen), underperforming CodeI/O at 3.5M samples.

Measured effects reported in the paper

Surprising findings

• Rejection sampling (discarding incorrect predictions) lowered performance, so the authors kept incorrect CoTs in the main training data.
• Replacing DeepSeek-V2.5 synthesis on WebInstruct improved it but still fell short of CodeI/O, suggesting gains are not only from distillation.

Limitations

• Data leakage concerns on LeetCode-O (21.5% 13-gram overlap) and KorBench, mitigated by a sample-wise analysis.
• Optimal data-mixing strategy between CodeI/O and instruction-tuning data left for future work.
• Second-turn revision yields minimal gains and sometimes regressions, so multi-turn revision was limited to one turn.

Experimental setup

Datasets: CodeMix PyEdu-R (Python-Edu subset) TheAlgorithms Python Project Euler LeetCode/Codewars/Edabit/Codeforces/AtCoder/CodeChef problems in-house instruction-tuning data WebInstruct OpenMathInstruct-2 OpenCoder-SFT-Stage-1 Tulu-3

Benchmarks: WinoGrande DROP GSM8K MATH GPQA MMLU-STEM LeetCode-O CRUXEval-I CRUXEval-O BBH-EN BBH-ZH ZebraLogic

Metrics: accuracy F1 average score

Baselines: WebInstruct OpenMathInstruct-2 OpenCoder-SFT-Stage-1 Python-Edu WebInstruct-DS25 Tulu-3

Builds on: CRUXEval Python-Edu rejection sampling

Closest papers by meaning

• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.92)
• CodePlan: Unlocking Reasoning Potential in Large Language Models by Scaling Code-form Planning (ICLR 2025 · similarity 0.90)
• Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.89)
• Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.89)
• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.89)
• Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization (NeurIPS 2025 · similarity 0.89)