Research map / Large language models / Agents, code and math
Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces code reasoning as a task spanning inductive, deductive, and abductive reasoning over eight benchmarks, and proposes RHDA, an iterative hypothesis decomposition and amendment pipeline with tool feedback that improves GPT-4o performance.
Problem
LLMs struggle with tasks that combine reasoning and recall, such as code reasoning (inducing, executing, or inverting programs), and existing prompting methods fail to find satisfactory reasoning pathways that generalize to unseen observations.
Approach
An iterative pipeline in which the LLM proposes a step-by-step hypothesis, decomposes it into sub-hypotheses, translates it into executable code, verifies it with an external tool (Python executor or compiler), and amends sub-hypotheses based on the feedback.
Key results
- On inductive code reasoning with GPT-4o, RHDA (T=2, N=3) reaches 48.80 task accuracy on List Function vs 38.00 for IO, and 43.06 accuracy on DeepCoder vs 37.85 for SC.
- On deductive reasoning with GPT-4o, RHDA (T=2, N=1) reaches 90.62 on CRUXEval and 84.16 on LiveCodeBench, versus 89.12 and 83.14 for CoT.
- On VirtualHome, RHDA reduces error actions from 92 (native GPT-4o) to 52 across 52 tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RHDA instead of the strongest listed baseline | GPT-4o on inductive code reasoning · List Function | task accuracy: 41.20 -> 48.80 | Helped |
| Use RHDA instead of the strongest listed baseline | GPT-4o on inductive code reasoning · MiniARC | task accuracy: 13.08 -> 13.85 | Helped |
| Use RHDA instead of the strongest listed baseline | GPT-4o on inductive code reasoning · RobustFill | task accuracy: 26.09 -> 34.78 | Helped |
| Use RHDA instead of the strongest listed baseline | GPT-4o on inductive code reasoning · DeepCoder | task accuracy: 26.04 -> 29.17 | Helped |
| Use RHDA instead of CoT | GPT-4o · CRUXEval (deductive) | accuracy: 89.12 -> 90.62 | Helped |
| Use RHDA instead of CoT | GPT-4o · LiveCodeBench (deductive) | accuracy: 83.14 -> 84.16 | Helped |
| Add hypothesis decomposition (ablation) | RHDA on inductive code reasoning · List Function | task accuracy: 33.20 -> 48.80 | Helped |
| Add hypothesis decomposition (ablation) | RHDA on inductive code reasoning · MiniARC | task accuracy: 4.62 -> 13.85 | Helped |
| Add hypothesis decomposition (ablation) | RHDA on inductive code reasoning · RobustFill | task accuracy: 21.74 -> 34.78 | Helped |
| Add hypothesis decomposition (ablation) | RHDA on inductive code reasoning · DeepCoder | task accuracy: 9.37 -> 29.17 | Helped |
| Add amendment reflection (ablation) | RHDA on deductive code reasoning · CRUXEval | accuracy: 86.62 -> 90.62 | Helped |
| Add amendment reflection (ablation) | RHDA on deductive code reasoning · LiveCodeBench | accuracy: 71.29 -> 84.16 | Helped |
Surprising findings
- IO prompting achieves higher per-observation accuracy than RHDA on inductive tasks, but lower task accuracy, because it does not produce a hypothesis satisfying all observations.
- Hypothesis decomposition alone (w/o Amend) and amendment alone (w/o Sub-Hyp) were each less effective than the full pipeline on several tasks; on deductive tasks, the sub-hypothesis-only variant sometimes underperformed CoT.
Limitations
- Sensitivity to initial hypotheses; amendments often fail to escape the original reasoning framework.
- Sub-hypothesis generation is often inaccurate for complex or abstract tasks.
- Hyperparameters (temperature, T, N) and prompts were not fully optimized; the work is a preliminary exploration.
Experimental setup
Datasets: List Function MiniARC RobustFill DeepCoder CRUXEval LiveCodeBench VirtualHome
Benchmarks: List Function MiniARC RobustFill DeepCoder CRUXEval LiveCodeBench VirtualHome
Metrics: accuracy task accuracy execution error rate API cost
Baselines: IO prompting Program of Thought Chain of Code Self-Consistency Self-Refine Standard prompting Chain of Thought
Builds on: chain-of-thought program of thought self-refine chain of code
Closest papers by meaning
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning (NeurIPS 2025 · similarity 0.92)
- RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation (ICML 2025 · similarity 0.91)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.90)
- Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle (NeurIPS 2024 · similarity 0.90)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.