Research map / Large language models / Language, safety and interpretability
Self-Infilling Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes self-infilling code generation, which uses the self-infilling ability of FIM-trained code LMs to add interruption and looping mechanisms to left-to-right decoding, improving code generation quality on HumanEval, MBPP, DS-1000, MultiPL-E and GSM8K without training.
Problem
Code language models trained with fill-in-the-middle are used almost exclusively left-to-right at inference, leaving their infilling ability unused for complete code generation, and left-to-right decoding is prone to error propagation and degenerate outputs.
Approach
FIM-trained models can generate a suffix from the prefix alone (self-infilling). The method interrupts left-to-right decoding when token confidence is low, generates a suffix anchored by a suffix prompt, then infills the middle; a looping mechanism alternates self-infilling and left-to-right passes to recurrently refine snippets.
Key results
- On HumanEval, Code Llama 7B with self-infilling (N=2) reaches 39.0 pass@1 vs 34.1 for left-to-right, and StarCoder reaches 38.4 vs 35.4.
- On DS-1000, self-infilling raises DeepSeek-Coder-Base 33B overall pass@1 from 42.8 to 43.7 and Code Llama 7B from 24.8 to 28.7.
- Self-infilling reduces degenerate outputs; for Code Llama 7B on HumanEval the degeneracy rate at tau=0.25 is 2.4%, versus about 13% at tau=0.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add self-infilling with 2 looping iterations instead of left-to-right decoding | Code Llama 7B · HumanEval zero-shot | pass@1: 34.1 -> 39.0 | Helped |
| Add self-infilling with 2 looping iterations instead of left-to-right decoding | Code Llama 7B · MBPP three-shot | pass@1: 42.8 -> 44.8 | Helped |
| Add self-infilling with 2 looping iterations instead of left-to-right decoding | Code Llama 13B · HumanEval zero-shot | pass@1: 35.4 -> 40.8 | Helped |
| Add self-infilling with 2 looping iterations instead of left-to-right decoding | Code Llama 13B · MBPP three-shot | pass@1: 47.2 -> 49.0 | Helped |
| Disable looping while retaining self-infilling interruption (ablation) | Code Llama 7B self-infilling · HumanEval zero-shot | pass@1: 34.1 -> 29.9 | Hurt |
| Use 1 looping iteration instead of 2 (ablation) | Code Llama 7B self-infilling · HumanEval zero-shot | pass@1: 39.0 -> 34.1 | Hurt |
| Use Vanilla instead of Extended suffix splitting (ablation) | Code Llama 7B looping · HumanEval | pass@1: 32.3 -> 37.2 | Helped |
| Use Half instead of Extended suffix splitting (ablation) | Code Llama 7B looping · HumanEval | pass@1: 32.3 -> 31.7 | Hurt |
| Use Vanilla instead of Extended suffix splitting (ablation) | Code Llama 7B looping · MBPP | pass@1: 43.8 -> 44.6 | Helped |
| Use Half instead of Extended suffix splitting (ablation) | Code Llama 7B looping · MBPP | pass@1: 43.8 -> 43.8 | No clear effect |
| Increase the interruption threshold from 0.1 to 0.25 (ablation) | Code Llama 7B self-infilling · HumanEval | degenerate output rate: nearly 13% -> 2.4% | Helped |
| Apply self-infilling to instruction-tuned models | Code Llama Instruct and DeepSeek-Coder Instruct · HumanEval and HumanEval+ | pass@1 | Mixed |
Surprising findings
- Interruption alone without looping (N=0) is slightly worse than left-to-right decoding on HumanEval and MBPP, and the looping mechanism is what yields gains.
- Self-infilling gives little or no benefit for instruction-tuned models (Code Llama - Instruct, DeepSeek-Coder-Instruct), attributed to missing infilling training in instruction tuning.
Limitations
- Looping incurs extra compute from repeated context processing and decoding.
- Gains are small or absent for StarCoder on DS-1000 and GSM8K and for instruct models, and the method is limited to single-span infilling by the FIM format.
Experimental setup
Datasets: HumanEval MBPP DS-1000 MultiPL-E GSM8K HumanEval+
Benchmarks: HumanEval MBPP DS-1000 MultiPL-E GSM8K HumanEval+
Metrics: pass@1 pass@10 pass@100 degeneracy rate
Baselines: Left-to-right decoding Insertion (DS-1000 infilling format) Code Llama - Instruct Code Llama - Python GPT-3.5 Turbo GPT-4 Turbo
Builds on: fill-in-the-middle nucleus sampling Gibbs sampling
Closest papers by meaning
- SelfCodeAlign: Self-Alignment for Code Generation (NeurIPS 2024 · similarity 0.87)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.85)
- Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation (NeurIPS 2024 · similarity 0.84)
- Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs (NeurIPS 2025 · similarity 0.84)
- Repoformer: Selective Retrieval for Repository-Level Code Completion (ICML 2024 · similarity 0.84)
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.