Research map / Large language models / Agents, code and math
DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds DomainEval, an automatically constructed multi-domain code generation benchmark (2454 subjects from GitHub repos, six domains) via test-method matching, and evaluates 12 LLMs showing strong domain gaps (computation high, cryptography and system low).
Problem
Existing code benchmarks like HumanEval and MBPP focus on common coding tasks, leaving LLMs' ability to generate domain-specific code (computation, network, system, cryptography, etc.) under-explored.
Approach
A fully automated, test-guided pipeline that traces tests in GitHub repositories back to function implementations via AST analysis, filters them for executability, significance and difficulty, and generates instructions with an LLM, yielding domain-partitioned benchmark subjects.
Key results
- Computation is the easiest domain with an average 82.44% Pass@1, while cryptography (33.08%) and system (37.50%) are the hardest.
- Llama-2-13b-chat shows the largest domain gap, 80.94% on computation versus 12.0% on cryptography.
- GPT-4o-mini has the highest average Pass@5 at 67.13%; Qwen2-72B-Instruct-GPTQ-Int4 is the best open-source model at 64.25%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase generated samples from 1 to 5 | LLMs evaluated on DomainEval · DomainEval overall | Pass@k mean: 53.42% to 59.60% | Helped |
| Increase generated samples from 1 to 5 | LLMs evaluated on DomainEval · DomainEval across six domains | Pass@k | Helped |
| Increase generated samples from 1 to 5 | LLMs evaluated on DomainEval · DomainEval domain-bias variation | standard deviation: 18.33 to 17.72 | No clear effect |
| Increase generated samples from 1 to 5 | CodeLlama-13b-Instruct · DomainEval domain-bias variation | standard deviation: 19.90 to 20.55 | Hurt |
| Fine-tune Llama-2-13b as CodeLlama-13b | Llama-2-13b · DomainEval overall | Pass@5 mean: 46.49% to 57.74% | Helped |
Surprising findings
- Increasing samples from N=1 to N=5 raised mean performance but did not reduce cross-domain deviation; CodeLlama-13b's deviation increased from 19.90 to 20.55.
- Fine-tuned CodeLlama-13b improves average score over Llama-2-13b by 11.25 points yet the domain gaps persist.
Limitations
- Restricted to Python functions from GitHub repositories with at least 100 stars and test functions that can be matched.
- Banned-keyword filtering removes file, process and network-related code for safety, which may bias domain coverage.
- Instructions are generated by an open LLM (Qwen2-72B), which may introduce description quality issues.
Experimental setup
Datasets: GitHub repositories with at least 100 stars
Benchmarks: DomainEval
Metrics: Pass@1 Pass@5 standard deviation across domains
Baselines: GPT-3.5-turbo GPT-4o-mini Qwen2-72B-Instruct DeepSeek-Coder-33b-instruct CodeLlama-34b-Instruct Llama-2-13b-chat
Builds on: HumanEval MBPP Pass@k
Closest papers by meaning
- EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations (NeurIPS 2024 · similarity 0.91)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.90)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.89)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.