Research map / Large language models / Agents, code and math
Is Code Better Than Language for Algorithmic Reasoning?
Terry Tong, Yu Feng, Surbhi Goel, Dan Roth
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| execute generated code with external Python runtime instead of LLM simulation (Route 2 -> Route 3) | code-trace reasoning · 40-task algorithmic set | accuracy: 17.37 -> 48.84 (+31.47pp) | Improved |
| use code traces instead of natural-language traces with LLM execution (Route 1 -> Route 2) | LLM reasoning · 40-task algorithmic set | accuracy: 17.21 -> 17.37 (+0.15pp) | No clear effect |
| add external execution to code traces for GPT-4o-mini | GPT-4o-mini · 40-task set, per model | accuracy: 12.9 -> 49.1 | Improved |
| use code traces instead of NL for GPT-4o-mini | GPT-4o-mini · 40-task set, per model | accuracy: 14.1 -> 12.9 | Hurt |
| use code traces instead of NL for Gemini 2.0 Flash | Gemini 2.0 Flash · 40-task set, per model | accuracy: 17.5 -> 18.1 | No clear effect |
| add external execution on O(n^3) and O(n log n) tasks | Route 3 · complexity strata | accuracy: 0.0 -> 0.0 | No clear effect |
| translate code traces to NL with more in-context shots (0 -> 5) (ablation) | translated NL trace, Claude Haiku 4.5 · Translation shot ablation (25% subset) | accuracy gain over question-only baseline: +9.70pp -> +17.30pp | Improved |
| use translated NL instead of native NL trace (5 shots) (ablation) | Claude Haiku 4.5 · Translation shot ablation (25% subset) | accuracy: 55.70% -> 50.00% | Hurt |
| add reasoning trace (native NL) to question-only prompt (ablation) | Claude Haiku 4.5 · Translation shot ablation (25% subset) | accuracy: 32.70% -> 55.70% | Improved |
| RLM execution: code arm instead of NL arm for Claude Haiku 4.5 | Claude Haiku 4.5 · RLM 25% subset, seed 0 | accuracy: 30.9% -> 29.4% | Hurt |
| RLM execution: code arm instead of NL arm for Gemini 2.5 Pro | Gemini 2.5 Pro · RLM 25% subset, seed 0 | accuracy: 45.2% -> 47.5% | Improved |
| code execution instead of simulation for Claude Opus 4.6 | Claude Opus 4.6 · Frontier controlled subset | accuracy: 73.42% -> 77.54% | Improved |
Closest papers by meaning
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.91)
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning (NeurIPS 2025 · similarity 0.89)
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.89)
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.88)
- Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.88)
- Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
