Research map / Large language models / Agents, code and math
How efficient is LLM-generated code? A rigorous & high-standard benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ENAMEL, a benchmark of 142 HumanEval/HumanEval+ problems with expert reference solutions and strong test generators, plus an unbiased Rao-Blackwellized eff@k metric handling right-censored execution times; evaluates 30 LLMs and finds they fall short of expert-level efficiency.
Problem
Existing code-generation benchmarks evaluate only correctness, and prior efficiency evaluations mishandle right-censored execution times, use weak canonical or random test cases, and use non-expert reference solutions, so they overestimate LLM code efficiency.
Approach
A pass@k generalization (eff@k) that takes the expected maximum of a continuous efficiency score which is zero whenever execution exceeds the time limit, with a Rao-Blackwellized bootstrap estimator, combined with expert-written optimal reference solutions, multi-level expert test generators, and correctness filtering.
Key results
- GPT-4 achieves eff@1=0.454 with pass@1=0.831 on ENAMEL, and no evaluated LLM reaches eff@1>0.5 under greedy decoding.
- eff@k is consistently much lower than pass@k across all 30 LLMs, model sizes, and sample sizes.
- Under the speedup metric, Mixtral 8x22B Instruct and Llama 3 70B Instruct appear to outperform GPT-4, a ranking reversal that eff@1 does not show.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of sampled code candidates from k=1 to k=100 | Phind Code Llama V2 · ENAMEL | eff@k: 0.351 -> 0.723 | Helped |
| Increase the number of sampled code candidates from k=1 to k=100 | Mixtral 8x22B Instruct · ENAMEL | eff@k: 0.407 -> 0.704 | Helped |
| Encourage generating the most efficient algorithm rather than use the basic prompt (ablation) | Llama 3 70B Instruct · ENAMEL | eff@1: 0.421 -> 0.418 | Hurt |
| Encourage generating the most efficient algorithm rather than use the basic prompt (ablation) | Mixtral 8x22B Instruct · ENAMEL | eff@1: 0.408 -> 0.426 | Helped |
| Increase timeout factor alpha from 1.5 to 3.5 (ablation) | ENAMEL evaluated on GPT-4 Turbo · ENAMEL | eff@1: 0.421 -> 0.541 | Helped |
| Increase level-1 hardness h1 from 1 to 5 (ablation) | ENAMEL evaluated on GPT-4 Turbo · ENAMEL | eff@1: 0.428 -> 0.498 | Helped |
| Increase level-3 hardness h3 from 1 to 5 (ablation) | ENAMEL evaluated on GPT-4 Turbo · ENAMEL | eff@1: 0.520 -> 0.460 | Hurt |
| Use the Rao--Blackwellized estimator instead of the vanilla estimator (ablation) | eff@k estimation with Llama 3 70B Instruct · ENAMEL, k=1 | standard deviation: 0.20 -> 0.02 | Helped |
| Use the Rao--Blackwellized estimator instead of the vanilla estimator (ablation) | eff@k estimation with Llama 3 70B Instruct · ENAMEL, k=10 | standard deviation: 0.25 -> 0.08 | Helped |
| Use expert-written test cases instead of random test cases (ablation) | Naive factorization algorithm · Problem #31, primality testing | eff@1: 0.91 -> 0.17 | Hurt |
| Use expert-written test cases instead of random test cases (ablation) | Fermat primality test · Problem #31, primality testing | eff@1: 1.25 -> 0.00 | Hurt |
| Evaluate with ENAMEL instead of HumanEval+ | Benchmark evaluation · HumanEval+ versus ENAMEL | eff@1: 0.513 -> 1.000 | Hurt |
Surprising findings
- Encouraging the model to produce the most efficient algorithm in the prompt barely improves efficiency (e.g., Llama 3 70B eff@1 0.421 to 0.418).
- GPT-4 Turbo has higher eff@1 than GPT-4 even though GPT-4 has higher pass@1.
Limitations
- Benchmark is limited to 142 HumanEval-derived problems in Python.
- Evaluation relies on a single expert's reference solutions and test generators, and some sampled models were evaluated with greedy decoding only due to cost.
- No error bars reported due to computational budget.
Experimental setup
Datasets: HumanEval HumanEval+
Benchmarks: ENAMEL
Metrics: eff@k pass@k speedup
Baselines: EffiBench Mercury speedup metric HumanEval canonical solutions HumanEval+ canonical solutions
Builds on: pass@k HumanEval HumanEval+ bootstrap Self-Refine
Closest papers by meaning
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.94)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.91)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.91)
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness (ICML 2024 · similarity 0.90)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.90)
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.