Research map / Large language models / Reasoning and chain of thought

Chain of Code: Reasoning with a Language Model-Augmented Code Emulator

ICML 2024 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Chain of Code, where an LM writes code and an interpreter executes lines when possible, falling back to the LM ("LMulator") to simulate outputs for non-executable semantic lines, evaluated on BIG-Bench Hard and other reasoning tasks.

Problem

Code-based reasoning helps arithmetic and algorithmic tasks but struggles on semantic reasoning tasks that are hard to express as executable code, and mixed semantic-numeric problems.

Approach

Line-by-line execution where a Python interpreter runs executable lines and an LM (the LMulator) simulates the output of lines that raise exceptions, updating a shared program state so code and language reasoning interleave.

Key results

• On BIG-Bench Hard with text-davinci-003, Chain of Code reaches 84% single-task accuracy versus 72% for CoT and 55% for direct prompting (human average 68%).
• With gpt-4 combined with CoC, the paper reports 91% on BIG-Bench Hard.
• Cross-task prompting on BBH with text-davinci-003 gives CoC 61% vs CoT 55% and direct 50%; CoC also beats direct prompting for smaller models where CoT does not.

Measured effects reported in the paper

Surprising findings

• Chain of Code outperforms direct prompting even for smaller models (ada, babbage, curie), whereas CoT only helps at the largest scale.
• Python-only execution fails almost entirely on semantic tasks (48% average vs 84% for interweaving).

Limitations

• Generating and interweaving code and LM execution adds context length and computation time.
• Code does not help on some semantic tasks such as Ruin Names.
• The LM cannot modify custom Python objects during simulated execution; program state is tracked as strings.

Experimental setup

Datasets: BIG-Bench Hard GSM8K BIG-Bench

Benchmarks: BIG-Bench Hard GSM8K

Metrics: accuracy

Baselines: Direct prompting Chain of Thought Program of Thoughts ScratchPad PaLM 2-S code variant

Builds on: Chain of Thought ScratchPad Program of Thoughts Program-aided language models

Closest papers by meaning

• Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.90)
• Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding? (ICML 2025 · similarity 0.89)
• Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs (ICML 2025 · similarity 0.89)
• To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.88)
• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.88)