Research map / Large language models / Agents, code and math
EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EffiBench-X, a multi-language (Python, C++, Java, JavaScript, Ruby, Go) benchmark of recent competitive programming problems that measures runtime, memory peak, and memory integral efficiency of LLM-generated code relative to human expert solutions, evaluating 26 LLMs.
Problem
Existing code efficiency benchmarks focus mainly on Python, reuse contaminated and simple problem sets, and do not measure efficiency across diverse programming languages.
Approach
Builds a multi-language efficiency benchmark from recent (post-October 2023) competitive programming problems paired with accepted expert solutions per language, with a sandboxed high-resolution profiler and clipped efficiency ratios (ET, MP, MI) relative to human solutions.
Key results
- Best model Qwen3-32B reaches ~62% average ET relative to human expert solutions (62.21%), while Gemini-2.5-Pro leads Pass@1 at 79.43% and MP at 75.60%.
- DeepSeek-R1 Python ET is 67.30% vs Java 52.23%, showing language-dependent efficiency gaps.
- Across 26 LLMs, efficiency scores (ET, MP, MI) generally remain well below 100%, with ET ranging from 9.69% (Gemma-3-4B-It) to 62.21% (Qwen3-32B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase Qwen3 model size from 8B to 32B | Qwen3 · EffiBench-X (six-language average) | Execution Time: 45.44% -> 62.21% | Helped |
| increase Qwen3 model size from 8B to 32B | Qwen3 · EffiBench-X (six-language average) | Pass@1: 53.50% -> 70.41% | Helped |
| increase Qwen2.5-Coder model size from 14B to 32B | Qwen2.5-Coder-Instruct · EffiBench-X (six-language average) | Execution Time: 32.06% -> 36.66% | Helped |
| increase Gemma-3 model size from 4B to 27B | Gemma-3-It · EffiBench-X (six-language average) | Execution Time: 9.69% -> 16.62% | Helped |
| use reasoning-enhanced variant instead of base model | Phi-4 · EffiBench-X (six-language average) | Execution Time: 28.42% -> 48.37% | Helped |
| use reasoning-enhanced variant instead of base model | Phi-4 · EffiBench-X (six-language average) | Pass@1: 30.60% -> 50.54% | Helped |
| use Gemini-2.0-Flash-Thinking instead of Gemini-2.0-Flash | Gemini-2.0 · EffiBench-X (six-language average) | Execution Time: 30.61% -> 38.82% | Helped |
| use Gemini-2.0-Flash-Thinking instead of Gemini-2.0-Flash | Gemini-2.0 · EffiBench-X (six-language average) | Pass@1: 48.56% -> 57.38% | Helped |
| use Python instead of Java | DeepSeek-R1 · EffiBench-X language subsets | Execution Time: 52.23% -> 67.30% | Helped |
| use Python instead of Java | Claude-3.7-Sonnet · EffiBench-X language subsets | Execution Time: 46.33% -> 51.90% | Helped |
| use functional problems instead of standard I/O problems | Qwen3-32B · EffiBench-X problem types | Execution Time: 49.25% -> 73.14% | Helped |
Surprising findings
- Qwen3-14B achieves the highest efficiency on the Java subset (62.32% ET), outperforming larger Qwen3-32B variants.
- Gemini-2.5-Pro has high correctness and memory peak but only 31.50% ET on the Go subset.
Limitations
- Focuses on competitive programming algorithmic efficiency; may not generalize to systems programming or large-scale data processing.
- Does not analyze root causes of inefficiency at code-construct level.
- Extensive multi-language, multi-model profiling is computationally demanding.
Experimental setup
Datasets: competitive programming problems from Aizu, AtCoder, CodeChef, Codeforces, LeetCode (released after October 2023)
Benchmarks: EffiBench-X
Metrics: Execution Time (ET) Memory Peak (MP) Memory Integral (MI) Pass@1
Baselines: EffiBench Mercury EvalPerf ENAMEL PIE COFFE
Builds on: EffiBench Mercury PIE HumanEval-X
Closest papers by meaning
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.94)
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.92)
- EffiBench: Benchmarking the Efficiency of Automatically Generated Code (NeurIPS 2024 · similarity 0.92)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.91)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.91)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.