Research map / Large language models / Agents, code and math
EffiBench: Benchmarking the Efficiency of Automatically Generated Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EffiBench, a 1,000-problem LeetCode-based benchmark with canonical human solutions and new normalized execution-time and memory metrics, used to show that code from 42 LLMs is substantially less efficient than canonical solutions.
Problem
Code generation benchmarks focus on correctness and rarely measure the efficiency (execution time and memory) of LLM-generated code, leaving efficiency gaps unexamined.
Approach
Pairs 1,000 efficiency-critical LeetCode problems with top-rated human canonical solutions and LLM-generated test-case generators, then measures normalized execution time, max memory, and total memory (area under memory-time curve) relative to canonical code.
Key results
- GPT-4 generated code averages 3.12x the execution time of canonical solutions, with worst-case execution time 13.89x and total memory 43.92x.
- StarCoder2-15B is the most efficient open-source model, at 2.59x NET, 1.71x NMU and 4.83x NTMU on average.
- Higher pass@1 does not imply higher efficiency; GPT-4-turbo-preview has higher pass@1 (65.4%) than GPT-4 but lower efficiency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate code with GPT-4 instead of using human-written canonical solutions | GPT-4 · EffiBench | NET: 1 -> 3.12 | Hurt |
| Generate code with GPT-4 instead of using human-written canonical solutions | GPT-4 · EffiBench | NTMU: 1 -> 6.36 | Hurt |
| Use StarCoder2-15B instead of human-written canonical solutions | StarCoder2-15B · EffiBench | NET: 1 -> 2.59 | Hurt |
| Use StarCoder2-15B instead of human-written canonical solutions | StarCoder2-15B · EffiBench | NTMU: 1 -> 4.83 | Hurt |
| Upgrade GPT-3.5-turbo from version 0301 to 1106 | GPT-3.5-turbo · EffiBench | NET: 3.18x -> 3.40x | Hurt |
| Generate code with OpenCodeInterpreter-DS-1.3B rather than the canonical solution | OpenCodeInterpreter-DS-1.3B · HumanEvalPlus | NET: 1 -> 0.86 | Helped |
Surprising findings
- Newer GPT-3.5-turbo version (1106) has higher pass@1 (49.3%) than 0301 (42.3%) yet its execution time ratio rose from 3.18x to 3.40x.
- On HumanEvalPlus, OpenCodeInterpreter-DS-1.3B generated code has NET 0.86, faster than canonical solutions.
Limitations
- Limited to Python.
- Only LeetCode algorithmic problems, excluding real-world coding scenarios.
- Efficiency measurements depend on hardware and software environment.
Experimental setup
Datasets: LeetCode problems (HuggingFace LeetCode collection) LeetCode discussion top-starred canonical solutions
Benchmarks: EffiBench HumanEvalPlus MBPPPlus
Metrics: execution time (ET) normalized execution time (NET) max memory usage (MU) normalized max memory usage (NMU) total memory usage (TMU) normalized total memory usage (NTMU) pass@1
Baselines: HumanEval MBPP APPS DS-1000 DSP HumanEvalPlus MBPPPlus Mercury
Builds on: MBPP DS-1000
Closest papers by meaning
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.92)
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.88)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.