Research map / Large language models / Agents, code and math
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PDB, a dataset-agnostic pipeline that injects verified atomic and multi-bug faults into coding datasets and scores debugging with edit-level precision and bug-level recall, showing frontier LLMs pass tests while over-editing.
Problem
Unit-test-only debugging evaluation cannot distinguish targeted minimal fixes from full regeneration or partial repairs, and existing debugging benchmarks are contamination-prone and costly to build.
Approach
Generate buggy programs by LLM-synthesizing unit-test-verified atomic bugs under ODC categories, compose independent bugs into multi-bug programs, and score patches with an epsilon-relaxed edit-level precision (after unit-test-based essential-edit minimization) and bug-level recall.
Key results
- Frontier models GPT-5.1-Codex and DeepSeek-V3.2-Thinking exceed 76% unit-test pass rate but have precision at or below 45% on PDB-Single (Qwen3-Coder-480B: 70% unit, 66% precision).
- On PDB-Wild, GPT-5.5 reaches 84.3% unit score and 83.0% recall but only 60.9% precision (tolerance epsilon=1).
- Iterative and agentic debugging improve unit-test pass rate and recall but do not improve precision; Claude-Code reaches only about 50% precision.
Surprising findings
- Freeform prompting drops precision substantially across models; GPT-5.1-Codex falls below 20% precision without minimal-edit instructions.
- Agentic feedback (unit tests and error messages) often fails to improve edit precision and can exacerbate regeneration.
Limitations
- Estimated 1.85% false-positive rate from inherited upstream test suites; test adequacy is not guaranteed.
- Atomicity cannot be strictly guaranteed for multi-line and repository-level blocks, so a looser tolerance epsilon=1 is used.
- Bugs are synthetic and generated by LLMs, so transfer to natural bugs is only partially validated.
Experimental setup
Datasets: BigCodeBench LiveCodeBench SWE-smith DebugBench
Benchmarks: PDB-Single PDB-Wild DebugBench
Metrics: edit-level precision bug-level recall unit-test pass@1
Baselines: GPT-5.1-Codex Claude-Sonnet-4.5 Gemini-2.5-Pro Qwen3-Coder-480B DeepSeek-V3.2-Thinking Kimi-K2-Thinking Grok-Code-Fast Claude-Code
Builds on: Orthogonal Defect Classification Pass@1
Closest papers by meaning
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.88)
- TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark (ICLR 2025 · similarity 0.86)
- LeDex: Training LLMs to Better Self-Debug and Explain Code (NeurIPS 2024 · similarity 0.85)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.85)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.85)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.