Research map / Large language models / Agents, code and math

Steering Large Language Models between Code Execution and Textual Reasoning

ICLR 2025 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates when LLMs choose code execution versus textual reasoning across 14 tasks and 6 LLMs, finds no single steering method is optimal, and proposes three methods (Code Interpreter+, Code+Text+Sum., Self-estimate Score) plus multi-turn refinement that improve performance.

Problem

LLMs do not reliably decide when to answer with code versus textual reasoning; existing steering methods (prompting, Code Interpreter, AutoGen) are inconsistent across tasks and models, and textual reasoning fails on math, logic, and planning tasks that code can solve.

Approach

Systematically benchmarks seven existing code/text steering methods across 14 code-solvable tasks and six LLMs, then proposes steering methods that combine a code-based answer and a text-based answer and have the same LLM summarize them, or let the model self-estimate its code and text confidence before choosing a mode, plus multi-turn code-execution refinement.

Key results

• Code+Text+Sum. achieves the highest average normalized score of 79.5 across six models, versus 77.2 for the best baseline (All Code), and ranks first on average rank (2.50).
• On GPT-4o, Code+Text+Sum. reaches 88.2 average normalized score versus 80.6 for Only Question; Code Interpreter+ lifts GPT-4o-mini Game 24 success from 62% to 83%.
• GPT-3.5 outperforms GPT-4o-mini and GPT-4o on several code-augmented tasks (e.g., Game 24 and Number Multiplying), an inverse scaling pattern; GPT-4o CI answered Number Multiplying with text at medium digit lengths where it failed.

Measured effects reported in the paper

Surprising findings

• Prompting LLMs to always answer with code performs as poorly or worse than always answering with text in Game 24, with code often mimicking textual reasoning.
• Smaller GPT models (GPT-3.5, GPT-4o-mini) with Code Interpreter outperform GPT-4o on some tasks, an inverse scaling behavior driven by GPT-4o's overconfidence in textual reasoning.

Limitations

• Results depend on the capability of the LLM; the proposed methods are less effective for GPT-3.5 and O1-preview.
• Combined code and text answers and multi-turn refinement raise token length and runtime costs substantially (e.g., 3679.8 tokens for Code+Text+Sum.).
• Multi-turn refinement degrades text-based answers and GPT-3.5 answers degrade across iterations.

Experimental setup

Datasets: Number Multiplying Game 24 GSM-Hard MATH-Geometry MATH-Count&Probability Date Understanding Web of Lies Logical Deduction Navigate BoxNet Path Plan Letters

Benchmarks: Number Multiplying Game 24 GSM-Hard MATH-Geometry MATH-Count&Probability Date Understanding Web of Lies Logical Deduction Navigate BoxNet Path Plan Letters

Metrics: success rate average normalized score token length runtime

Baselines: Only Question All Text All Code All Code + CoT AutoGen Conca. AutoGen System Code Interpreter

Builds on: AutoGen Chain-of-Thought Tree-of-Thought BIG-Bench-Hard

Closest papers by meaning

• CodeSteer: Symbolic-Augmented Language Models via Code/Text Guidance (ICML 2025 · similarity 0.90)
• The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.90)
• What Makes Large Language Models Reason in (Multi-Turn) Code Generation? (ICLR 2025 · similarity 0.89)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.89)
• Fleet of Agents: Coordinated Problem Solving with Large Language Models (ICML 2025 · similarity 0.89)
• EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.89)