Research map / Large language models / Agents, code and math
TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TestGenEval, a benchmark built from SWE-bench repositories for file-level unit test suite generation and test completion, scored with pass@k, coverage and mutation score, and evaluates 9 open and closed LLMs.
Problem
Existing test generation benchmarks are small, self-contained, use equivalence tests rather than human-written tests, rarely report coverage or mutation score, and do not evaluate test completion on real repositories.
Approach
Adapts SWE-bench Docker environments of 11 large Python repositories into file-level test generation and test completion tasks (first, last, extra tests), with human-written gold tests, and scores generated tests by coverage and mutation score (cosmic-ray) in addition to pass rates.
Key results
- GPT-4o achieves the highest average coverage of 35.2% and mutation score of 18.8% on full test suite generation.
- Codestral 22B achieves pass@5 of 74.3% on last test completion, the best among models.
- Models add near-zero coverage in the last and extra test completion settings (at most 0.5%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o rather than Llama 3.1 405B for full test suite generation | Full test suite generation · TestGenEval | coverage: 35.0% -> 35.2% | Helped |
| Use GPT-4o rather than Llama 3.1 405B for full test suite generation | Full test suite generation · TestGenEval | mutation score: 16.4% -> 18.8% | Helped |
| Increase model size within the Llama 3.1 family from 8B to 70B | Llama 3.1 test generation · TestGenEval full test generation | Any Pass@1: 30.5% -> 60.7% | Helped |
| Increase model size within the Gemma family from 9B to 27B | Gemma test generation · TestGenEval full test generation | Any Pass@1: 42.1% -> 57.7% | Helped |
| Provide more of the test file as context | Test completion · TestGenEval | Pass@5 | Helped |
| Increase context length for test completion up to around 32k tokens | Test completion · TestGenEval | Pass@5 | Helped |
| Increase context length for full test generation | Full test generation · TestGenEval | coverage | No clear effect |
| Sample more candidate test suites | Llama 3.1 8B · TestGenEval full test generation | coverage@k | Helped |
| Sample more test completions, particularly the first five | Llama 3.1 8B · TestGenEval test completion | Pass@k | Helped |
| Generate the last or extra test rather than the first test | Test completion · TestGenEval | coverage improvement | Hurt |
| Use Codestral 22B rather than other evaluated models for last-test completion | Test completion · TestGenEval | Pass@5: 74.3% | Helped |
Surprising findings
- Test completion for the last test adds almost no coverage (at most 0.5%) even for top models, since models mostly retest already covered paths.
- Llama 3.1 405B and GPT-4o rank differently on all pass@1 versus coverage, as verbose models are penalized by all pass@1.
Limitations
- Mutation scoring is computationally expensive, with 20% of files timing out at one hour.
- Benchmark is built on SWE-bench repositories, risking overfitting to that distribution.
- Data contamination risk is only partially assessed via perplexity on 10 tests.
Experimental setup
Datasets: SWE-bench
Benchmarks: TestGenEval TestGenEvalLite
Metrics: all pass@1 any pass@1 coverage coverage@pass mutation score mutation score@pass pass@1 pass@5 coverage improvement
Baselines: TestEval HumanEval SWT-Bench R2E
Builds on: SWE-bench TestEval R2E cosmic-ray
Closest papers by meaning
- SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents (NeurIPS 2024 · similarity 0.90)
- Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? (NeurIPS 2026 · similarity 0.86)
- DPIAgent: Divide, Protocol, Isolate for Agentic Reproduction Test Generation (NeurIPS 2026 · similarity 0.86)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.85)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.85)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.