Research map / Large language models / Agents, code and math
EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EvoCodeBench, an evolving repo-level Python code generation benchmark with domain labels and domain-specific metrics (DSI), releasing its first version EvoCodeBench-2403 (275 samples) and evaluating 8 LLMs.
Problem
Existing code generation benchmarks suffer from data leakage into LLM training data and lack domain-specific evaluation, so practitioners cannot choose LLMs for particular programming domains.
Approach
A periodically updated repo-level benchmark built from recently created GitHub repositories with LLM-generated requirements and domain labels from a PyPI-derived 10-domain taxonomy, plus a Domain-Specific Improvement metric that flags comfort and strange domains per model.
Key results
- The highest Pass@1 of gpt-4 on EvoCodeBench-2403 is only 20.73% (infilling setting), versus 53.04 reported on a prior repo-level benchmark.
- Leak ratio on EvoCodeBench-2403 drops to at most 2.18% (gpt-4) compared with 41.47% for gpt-3.5 on HumanEval under CDD detection.
- gpt-4 has the most comfort domains (e.g., Text Processing DSI 100%) but falls behind in Internet (DSI -28.59%); StarCoder 2-15B performs well in Database, outperforming 33B models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Local File (Completion) code context instead of no context | gpt-4 · EvoCodeBench-2403 | Pass@1: 7.27 -> 17.45 | Helped |
| Use Local File (Infilling) code context instead of no context | gpt-4 · EvoCodeBench-2403 | Pass@1: 7.27 -> 20.73 | Helped |
| Retrieve similar functions as context | gpt-4 · EvoCodeBench-2403 | Pass@1: 8.31 -> 12.29 | Helped |
| Retrieve similar functions as context | gpt-3.5 · EvoCodeBench-2403 | Pass@1: 6.64 -> 11.62 | Helped |
| Use recent-repository EvoCodeBench-2403 instead of HumanEval | gpt-3.5 leakage detection · HumanEval vs EvoCodeBench-2403 | Leak Ratio (%): 41.47 -> 1.75 | Helped |
| Choose DeepSeek Coder 6.7B instead of StarCoder 2 7B | model selection · EvoCodeBench-2403, Internet | Pass@1: 26.67 vs 26.67 | No clear effect |
| Evaluate models in the Internet domain rather than by overall Pass@1 | StarCoder 2 7B vs DeepSeek Coder 6.7B · EvoCodeBench-2403, Internet | Pass@1: 13.82 > 13.45; 26.67 vs 26.67 | Mixed |
| Select StarCoder 2 15B for Database tasks | StarCoder 2 15B · EvoCodeBench-2403, Database | Pass@1: 38.89 | Helped |
Surprising findings
- StarCoder 2-15B unexpectedly performs well in the Database domain and outperforms 33B LLMs.
- gpt-4 performs worse than other models in the Internet domain despite leading in most domains.
Limitations
- Python-only benchmark; no other programming languages.
- Small size (275 samples) and unbalanced domain distribution (e.g., Security has 1 sample).
- Auto-generated requirements and domain labels by gpt-4 can miss details or be wrong in a small fraction of cases.
Experimental setup
Datasets: EvoCodeBench-2403 (collected from GitHub repositories created Oct 2023 - Mar 2024)
Benchmarks: EvoCodeBench-2403
Metrics: Pass@k Recall@k Domain-Specific Improvement (DSI) leak ratio
Baselines: HumanEval ClassEval DevEval CoderEval
Builds on: HumanEval CoderEval DevEval CDD
Closest papers by meaning
- DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation (AAAI 2025 · similarity 0.91)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.88)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.