Research map / Large language models / Agents, code and math
NExT: Teaching Large Language Models to Reason about Code Execution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NExT, a self-training method that finetunes PaLM 2-L to generate natural-language rationales grounded in inline-formatted execution traces for program repair, filtered by unit tests.
Problem
LLMs trained on code surface text lack semantic understanding of program execution, and CoT prompting with execution traces in natural language is unreliable for program repair.
Approach
Iterative expert-iteration self-training where the model is given inline-commented execution traces as input and learns to produce natural-language rationales plus fixes, with unit-test results as the weak-supervision filter; training focuses on hard problems and always finetunes from the initial checkpoint.
Key results
- NExT improves PaLM 2-L end-to-end pass@1 on MBPPR from 23.2 to 49.3 (+26.1 absolute).
- On HumanEvalPack, NExT gains +14.3 end-to-end fix rate over PaLM 2-L without training data from that benchmark (42.5 vs 32.2 pass@1 greedy-to-sampling comparison as reported).
- Human annotation: NExT rationales judged more often correct than PaLM 2-L (overall 50.5% vs 34.9%), comparable to GPT-3.5 (51.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply NExT self-training with rationales and execution traces | PaLM 2-L · MBPP-R | pass@1: 23.2 -> 49.3 | Helped |
| apply NExT self-training with rationales and execution traces | PaLM 2-L · MBPP-R | pass@25: 65.0 -> 79.4 | Helped |
| apply NExT self-training with rationales and execution traces | PaLM 2-L · HumanEvalPack | pass@1: 32.2 -> 42.5 | Helped |
| add natural-language rationales during iterative training (ablation) | NExT on PaLM 2-L · MBPP-R | pass@25: 71.8 -> 79.4 | Helped |
| add natural-language rationales during iterative training (ablation) | NExT on PaLM 2-L · HumanEvalPack | pass@25: 69.4 -> 76.5 | Helped |
| add execution traces during training (ablation) | NExT on PaLM 2-L · MBPP-R | pass@1: 44.1 -> 49.3 | Helped |
| add execution traces during training (ablation) | NExT on PaLM 2-L · HumanEvalPack | pass@1: 38.1 -> 42.5 | Helped |
| train only on hard examples instead of both easy and hard examples (ablation) | NExT on PaLM 2-L · MBPP-R | pass@25: 73.4 -> 79.4 | Helped |
| train only on hard examples instead of both easy and hard examples (ablation) | NExT on PaLM 2-L · HumanEvalPack | pass@25: 76.6 -> 76.5 | No clear effect |
| remove traces from the input at test time (ablation) | NExT trained with traces · MBPP-R | pass@1: 49.3 -> 40.8 | Hurt |
| add execution traces to few-shot CoT prompting (ablation) | PaLM 2-L · MBPP-R | pass@1: 16.3 -> 23.2 | Helped |
| add execution traces to few-shot CoT prompting (ablation) | PaLM 2-L · HumanEvalPack | pass@1: 30.3 -> 32.2 | Helped |
Surprising findings
- CoT prompting with traces is worse than vanilla prompting on HumanEvalPack for PaLM 2-L and code models (average drop of 7.8%), indicating LLMs struggle to reason with traces in NL out of the box.
- Removing traces at test time, NExT still reaches 40.8% pass@1 on MBPPR, only 3.3 points below the in-distribution result.
Limitations
- Traces are limited to native Python and to variable states, and do not capture if-condition outcomes or complex objects such as tensors.
- Rationale quality is approximated by proxy-based evaluation and a limited human study of 104 tasks; fix suggestions remain weaker than GPT-3.5's.
- MBPPR test filtering uses all tests for training and evaluation, which can allow overfitting fixes to tests.
Experimental setup
Datasets: MBPP-Repair (MBPPR, newly constructed from MBPP) HumanEvalFix augmented with EvalPlus tests (HumanEvalPack)
Benchmarks: MBPPR HumanEvalPack (HumanEvalFix with EvalPlus tests)
Metrics: pass@k end-to-end fix rate proxy-based pass@k using smaller LMs human rating of rationale quality (explain bugs, suggest fix
Baselines: PaLM 2-L 3-shot GPT-4 GPT-3.5 Mixtral-8x7B DeepSeek Coder 33B StarCoder SelfAPR-style NExT w/o rationale NExT w/o traces
Builds on: STaR SelfAPR Scratchpad Self-Debugging chain-of-thought prompting
Closest papers by meaning
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.90)
- SemCoder: Training Code Language Models with Comprehensive Semantics Reasoning (NeurIPS 2024 · similarity 0.90)
- Chain of Execution Supervision Promotes General Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- LeDex: Training LLMs to Better Self-Debug and Explain Code (NeurIPS 2024 · similarity 0.88)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.88)
- Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.