Research map / Large language models / Agents, code and math
Execution Guided Line-by-Line Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EG-CFG, an inference-time method that samples line-level code continuations, executes them against tests, and injects execution traces into the prompt via classifier-free guidance, reaching SOTA on several code benchmarks with DeepSeek models.
Problem
LLM code generation typically ignores execution feedback during inference; existing feedback methods operate in coarse, sequential generate-execute-refine cycles rather than token-level guidance.
Approach
Line-by-line generation where beam-sampled candidate continuations are executed, their execution traces are injected into the prompt, and classifier-free guidance interpolates between the unconditional solution prompt and the execution-conditioned prompt; multiple configured agents run in parallel on each task.
Key results
- With DeepSeek-V3-0324, EG-CFG reaches 96.6% on MBPP and 73.0% on MBPP-ET, above the reported QualityFlow (94.2%) and LPW results.
- On HumanEval, EG-CFG matches LDB at 99.4% and reaches 89.02% on HumanEval-ET.
- On DS-1000 EG-CFG reaches 69.9% and on CodeContests 60.6% with DeepSeek-V3-0324; with DeepSeek-Coder-1.3B it reaches 83.2% on MBPP versus 49.4% baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · MBPP | accuracy: 82.8 -> 96.6 | Helped |
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · MBPP-ET | accuracy: 64.8 -> 73.0 | Helped |
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · HumanEval | accuracy: 82.92 -> 99.4 | Helped |
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · HumanEval-ET | accuracy: 79.20 -> 89.02 | Helped |
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · DS-1000 | accuracy: 38.9 -> 69.9 | Helped |
| add line-by-line execution-guided CFG generation | DeepSeek-V3-0324 · CodeContests | accuracy: 41.81 -> 60.6 | Helped |
| add beam search to generate multiple candidate continuations (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP | accuracy: 58.2 -> 83.2 | Helped |
| add beam search to generate multiple candidate continuations (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP-ET | accuracy: 43.6 -> 59.8 | Helped |
| use CFG guidance rather than the gamma=1 ablation (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP | accuracy: 75.2 -> 83.2 | Helped |
| use CFG guidance rather than the gamma=1 ablation (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP-ET | accuracy: 48.2 -> 59.8 | Helped |
| use detailed execution traces instead of the final event only (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP | accuracy: 76.4 -> 83.2 | Helped |
| use detailed execution traces instead of the final event only (ablation) | EG-CFG on DeepSeek-Coder 1.3b · MBPP-ET | accuracy: 51.2 -> 59.8 | Helped |
Surprising findings
- Removing beam search drops DeepSeek-Coder-1.3B MBPP accuracy from 83.2% to 58.2%, close to baseline.
- Raising MGDebugger and MapCoder retry counts 40x yielded only modest gains (to 93.6% and 88.8%), below EG-CFG's 96.6%.
Limitations
- Added inference overhead from beam search, candidate execution, and dual-distribution CFG.
- Requires executable test cases that adequately exercise the target functionality.
- Bottom-up inference loop does not exploit task decomposition.
Experimental setup
Datasets: MBPP MBPP-ET HumanEval HumanEval-ET DS-1000 CodeContests
Benchmarks: MBPP MBPP-ET HumanEval HumanEval-ET DS-1000 CodeContests
Metrics: accuracy (pass all tests) relative success rate (RSR)
Baselines: Baseline LLM MGDebugger MapCoder QualityFlow LPW CodeSim LDB CONLINE SelfEvolve
Builds on: Classifier-Free Guidance Self-Debugging LPW MapCoder MGDebugger
Closest papers by meaning
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.88)
- Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking (NeurIPS 2025 · similarity 0.86)
- Stay on Topic with Classifier-Free Guidance (ICML 2024 · similarity 0.86)
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.86)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.86)
- Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.