Research map / Large language models / Agents, code and math
Is Self-Repair a Silver Bullet for Code Generation?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes whether LLM self-repair (debug and fix own code with feedback) improves code generation on HumanEval and APPS once sampling cost is accounted for, and tests the role of feedback quality via stronger-model and human feedback.
Problem
Self-repair is popular for boosting LLM code generation, but its benefit relative to simply drawing more i.i.d. samples at equal compute is poorly understood, especially the role of the textual feedback stage.
Approach
Models self-repair as a repair tree (generate programs, execute tests, generate feedback, repair) and compares pass rates against i.i.d. sampling at matched sample budgets; isolates feedback quality by swapping in feedback from stronger models or human programmers.
Key results
- On APPS with GPT-4, 10 initial samples plus 1 repair each gives a pass rate 1.05x that of pass@20 without repair; 2 initial samples with 10 repairs each gives 0.97x of pass@22.
- Replacing GPT-4's own feedback with human-written feedback raises the repair success rate from 33.3% to 52.6% (1.58x).
- Boosting feedback with a stronger model (GPT-4 feedback for GPT-3.5 on APPS, GPT-3.5 feedback for Code Llama) beats both the i.i.d. baseline and self-repair at all budgets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GPT-4's feedback with human feedback (ablation) | GPT-4 repair · APPS overall | repair success rate: 33.30% -> 52.60%; 1.58× | Helped |
| Use GPT-3.5 feedback instead of Code Llama feedback (ablation) | Code Llama repair · HumanEval | repair success rate: 9.1% -> 20.1% | Helped |
| Use GPT-4 feedback instead of Code Llama feedback (ablation) | Code Llama repair · HumanEval | repair success rate: 9.1% -> 39.3% | Helped |
| Use GPT-4 feedback instead of GPT-3.5 feedback (ablation) | GPT-3.5 repair · APPS overall | repair success rate: 4.7% -> 11.5% | Helped |
| Use GPT-4 feedback instead of GPT-3.5 feedback (ablation) | GPT-3.5 repair · APPS competition | repair success rate: 1.5% -> 3.3% | Helped |
| Use GPT-3.5 feedback instead of Code Llama feedback (ablation) | Code Llama repair · APPS overall | repair success rate: 1.1% -> 2.2% | Helped |
| Increase initial program samples while holding feedback-repair samples fixed (ablation) | Self-repair · HumanEval and APPS | pass rate relative to equal-budget i.i.d. sampling | Helped |
| Increase feedback-repair samples while holding initial samples fixed (ablation) | Self-repair · HumanEval and APPS | pass rate relative to equal-budget i.i.d. sampling | Mixed |
| Allocate 10 initial samples and 1 repair candidate each rather than 2 initial samples and 10 repair candidates each (ablation) | GPT-4 self-repair · APPS | pass rate relative to equal-budget i.i.d. sampling: 1.05×; 0.97× | Mixed |
| Use self-repair rather than i.i.d. sampling, accounting for equal sample budget | GPT-4 · APPS | pass rate relative to baseline: up to 8% | Helped |
| Use self-repair rather than i.i.d. sampling, accounting for equal sample budget | GPT-3.5 · APPS competition | pass rate relative to baseline: up to a 34% performance gain | Helped |
| Use self-repair rather than i.i.d. sampling, accounting for equal sample budget | Code Llama · HumanEval | pass rate relative to baseline: up to 10% | Helped |
Surprising findings
- Self-repair often fails to beat i.i.d. sampling at matched budgets, especially at small budgets.
- Increasing the number of repair samples per program generally gives little benefit; diversity of initial samples matters more.
Limitations
- Experiments limited to self-contained Python tasks with executable unit tests, unlike real-world software development.
- Bootstrapping from a single large repair tree may introduce statistical artefacts.
- Human study is small (16 participants, 40 programs) and did not track debugging time.
Experimental setup
Datasets: HumanEval APPS
Benchmarks: HumanEval APPS
Metrics: pass rate pass@k normalized pass rate repair success rate pass@t
Baselines: i.i.d. sampling without repair pass@k baseline
Builds on: pass@k Self-Debugging
Closest papers by meaning
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.90)
- AuPair: Golden Example Pairs for Code Repair (ICML 2025 · similarity 0.88)
- Code Repair with LLMs gives an Exploration-Exploitation Tradeoff (NeurIPS 2024 · similarity 0.86)
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.86)
- NExT: Teaching Large Language Models to Reason about Code Execution (ICML 2024 · similarity 0.86)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.