Research map / Large language models / Agents, code and math
Large Language Models of Code Fail at Completing Code with Potential Bugs
Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, George Karypis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace reference partial code with a prefix containing a potential bug | CodeGen-2B-mono · buggy-HumanEval | pass@1: 54.9 -> 3.1 | Hurt |
| replace reference partial code with a prefix containing a potential bug | CodeGen-2B-mono · buggy-FixEval | pass@1: 37.8 -> 4.3 | Hurt |
| remove buggy partial code before completion | CodeGen-2B-mono · buggy-HumanEval | pass@1: 3.1 -> 9.3 | Improved |
| rewrite the likely buggy line before completion | CodeGen-2B-mono · buggy-HumanEval | pass@1: 3.1 -> 24.9 | Improved |
| complete first, then apply program repair | CodeGen-2B-mono · buggy-HumanEval | pass@1: 3.1 -> 23.6 | Improved |
| remove buggy partial code before completion | CodeGen-2B-mono · buggy-FixEval | pass@1: 4.3 -> 8.6 | Improved |
| rewrite the likely buggy line before completion | CodeGen-2B-mono · buggy-FixEval | pass@1: 4.3 -> 7.2 | Improved |
| complete first, then apply program repair | CodeGen-2B-mono · buggy-FixEval | pass@1: 4.3 -> 4.7 | Improved |
| apply rewriting-then-completion to clean reference prefixes (ablation) | CodeGen-2B-mono · buggy-HumanEval reference setting | pass@1: 54.9 -> 49.6 | Hurt |
| apply completion-then-rewriting to clean reference prefixes (ablation) | CodeGen-2B-mono · buggy-HumanEval reference setting | pass@1: 54.9 -> 27.7 | Hurt |
| set bug-detection threshold to 0.9 rather than 1 (ablation) | Rewriting-then-completion with CodeGen-2B-mono · buggy-FixEval reference prefixes | pass@1: 37.8 -> 37.0 | No clear effect |
| set bug-detection threshold to 0.9 rather than 1 (ablation) | Rewriting-then-completion with CodeGen-2B-mono · buggy-FixEval buggy prefixes | pass@1: 4.3 -> 7.2 | Improved |
Closest papers by meaning
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.86)
- CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs (ICLR 2025 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completion (ICML 2026 · similarity 0.86)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
