AI Paper Atlas

Research map / Large language models / Agents, code and math

Large Language Models of Code Fail at Completing Code with Potential Bugs

Tuan Dinh, Jinman Zhao, Samson Tan, Renato Negrinho, Leonard Lausen, Sheng Zha, George Karypis

NeurIPS 2023 · Large language models › Agents, code and math · approach: Execution-based code verification · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace reference partial code with a prefix containing a potential bugCodeGen-2B-mono · buggy-HumanEvalpass@1: 54.9 -> 3.1Hurt
replace reference partial code with a prefix containing a potential bugCodeGen-2B-mono · buggy-FixEvalpass@1: 37.8 -> 4.3Hurt
remove buggy partial code before completionCodeGen-2B-mono · buggy-HumanEvalpass@1: 3.1 -> 9.3Improved
rewrite the likely buggy line before completionCodeGen-2B-mono · buggy-HumanEvalpass@1: 3.1 -> 24.9Improved
complete first, then apply program repairCodeGen-2B-mono · buggy-HumanEvalpass@1: 3.1 -> 23.6Improved
remove buggy partial code before completionCodeGen-2B-mono · buggy-FixEvalpass@1: 4.3 -> 8.6Improved
rewrite the likely buggy line before completionCodeGen-2B-mono · buggy-FixEvalpass@1: 4.3 -> 7.2Improved
complete first, then apply program repairCodeGen-2B-mono · buggy-FixEvalpass@1: 4.3 -> 4.7Improved
apply rewriting-then-completion to clean reference prefixes (ablation)CodeGen-2B-mono · buggy-HumanEval reference settingpass@1: 54.9 -> 49.6Hurt
apply completion-then-rewriting to clean reference prefixes (ablation)CodeGen-2B-mono · buggy-HumanEval reference settingpass@1: 54.9 -> 27.7Hurt
set bug-detection threshold to 0.9 rather than 1 (ablation)Rewriting-then-completion with CodeGen-2B-mono · buggy-FixEval reference prefixespass@1: 37.8 -> 37.0No clear effect
set bug-detection threshold to 0.9 rather than 1 (ablation)Rewriting-then-completion with CodeGen-2B-mono · buggy-FixEval buggy prefixespass@1: 4.3 -> 7.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy