Research map / Large language models / Language, safety and interpretability
Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAFIM, a syntax-aware fill-in-the-middle benchmark of 17,720 executable/syntactic code completion tasks, and evaluates 15+ LLMs with multiple prompt designs and a syntax-aware truncation method.
Problem
Existing fill-in-the-middle (FIM) code benchmarks such as HumanEval-Infilling are tiny, Python-only, and randomly masked; comparisons across models are also skewed by prompt choice and output truncation, and possible data contamination threatens validity.
Approach
Mask syntactically meaningful AST structures (algorithmic blocks, control-flow conditions, API calls) from post-April-2022 code, evaluate with unit-test execution or syntax matching, and use five prompt designs plus an AST-based syntax-aware truncation so that FIM and non-FIM models are compared fairly.
Key results
- DeepSeekCoder-33B achieves the best average Pass@1 of 69.0% on SAFIM, while StarCoder (15.5B, 55.5%) is comparable to GPT-4 (53.3%).
- Syntax-aware truncation raises CodeLLaMa-13B's algorithmic-block Pass@1 from 16.4% to 41.4%, changing its ranking relative to InCoder-6B.
- FIM-pretrained StarCoder outperforms L2R-only CodeGen-16B in L2R mode despite similar size, suggesting FIM pretraining does not harm and may improve L2R performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add syntax-aware truncation (ablation) | CodeGen-16B · SAFIM algorithmic block completion | Pass@1: 0.0 -> 25.9 | Helped |
| Add syntax-aware truncation (ablation) | CodeLLaMa-13B · SAFIM algorithmic block completion | Pass@1: 16.4 -> 41.4 | Helped |
| Add syntax-aware truncation (ablation) | CodeLLaMa-13B · SAFIM control-flow completion | Pass@1: 27.8 -> 57.2 | Helped |
| Add syntax-aware truncation (ablation) | GPT-3.5 (175B) · SAFIM control-flow completion | Pass@1: 37.4 -> 37.5 | No clear effect |
| Add syntax-aware truncation (ablation) | GPT-3.5 (175B) · SAFIM API function call completion | Pass@1: 44.2 -> 53.9 | Helped |
| Use the SPM prompt instead of IPF (ablation) | CodeGen-16B · SAFIM algorithmic block completion | Pass@1: 15.2 -> 25.9 | Helped |
| Use the PSM prompt instead of IPF (ablation) | DeepSeekCoder-33B · SAFIM algorithmic block completion | Pass@1: 33.8 -> 60.8 | Helped |
| Use the 1S prompt instead of L2R (ablation) | GPT-3.5 (175B) · SAFIM algorithmic block completion | Pass@1: 23.2 -> 31.2 | Helped |
Surprising findings
- Model size matters less than pretraining method and data: the weakest CodeLLaMa surpasses the strongest CodeGen by 14 points, and a 15.5B StarCoder matches GPT-4 on average.
- Without syntax-aware truncation, CodeGen-16B scores 0.0% on algorithmic blocks because outputs are unbounded.
Limitations
- Cross-family comparisons are not controlled experiments since pretraining environments differ.
- Possible data contamination for CodeLLaMa and DeepSeekCoder whose cutoffs overlap the benchmark window.
- API split is small (310 examples) and uses syntax matching rather than execution.
Experimental setup
Datasets: Codeforces solutions and unit tests GitHub code files via GH Archive
Benchmarks: SAFIM
Metrics: Pass@1 compile error rate (CErr%)
Baselines: GPT-3.5 GPT-4 CodeGen InCoder CodeLLaMa StarCoder DeepSeekCoder
Builds on: HumanEval-Infilling InCoder ExecEval CodeBLEU
Closest papers by meaning
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.90)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.89)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.89)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.88)
- Active Evaluation Acquisition for Efficient LLM Benchmarking (ICML 2025 · similarity 0.88)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.