Research map / Large language models / Instruction and fine-tuning
Mercury: A Code Efficiency Benchmark for Code Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Mercury, a 1,889-task LeetCode-based Python benchmark with runtime-distribution-based efficiency metric Beyond, and shows DPO beats SFT for improving code efficiency of code LLMs.
Problem
Code LLM benchmarks evaluate only functional correctness and ignore computational efficiency of generated code, and have limited test coverage and task difficulty.
Approach
Normalize code runtime against a per-task distribution of historical human solutions to get a runtime-percentile metric (Beyond) that jointly reflects correctness (failed solutions score 0) and efficiency; pair with LLM-generated test case generators and LoRA-based SFT/DPO baselines.
Key results
- Leading Code LLMs reach about 65% Pass on Mercury but under 50% Beyond, indicating a notable efficiency gap.
- DPO improves Beyond over base models for models above 6.7B, e.g., deepseek-coder-33b-base Beyond rises from 48.53 to 66.47 overall with DPO, while SFT lowers it to 38.32.
- SFT reduces Beyond for most models and lowers Pass for the largest models (e.g., deepseek-coder-33b SFT Pass 58.7 vs base 65.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DPO on Mercury-train | deepseek-coder-33b-base · Mercury-eval | Beyond: 48.53 -> 66.47 | Helped |
| Apply DPO on Mercury-train | CodeLlama-34b-hf · Mercury-eval | Beyond: 42.40 -> 63.94 | Helped |
| Apply DPO on Mercury-train | starcoder2-15b · Mercury-eval | Beyond: 49.17 -> 58.95 | Helped |
| Apply DPO on Mercury-train | CodeLlama-13b-hf · Mercury-eval | Beyond: 38.01 -> 47.39 | Helped |
| Apply DPO on Mercury-train | starcoder2-3b · Mercury-eval | Beyond: 33.40 -> 32.42 | Hurt |
| Apply DPO on Mercury-train | CodeLlama-7b-hf · Mercury-eval | Beyond: 27.45 -> 33.29 | Helped |
| Apply SFT on Mercury-train | starcoder2-7b · Mercury-eval | Beyond: 40.37 -> 35.61 | Hurt |
| Apply SFT on Mercury-train | deepseek-coder-33b-base · Mercury-eval | Beyond: 48.53 -> 38.32 | Hurt |
| Apply SFT on Mercury-train | CodeLlama-34b-hf · Mercury-eval | Beyond: 42.40 -> 36.91 | Hurt |
| Apply DPO on Mercury-train | CodeQwen1.5-7B · Mercury-eval | Beyond: 47.78 -> 48.52 | Helped |
| Apply DPO on Mercury-train | deepseek-coder-1.3b-base · Mercury-eval | Beyond: 35.62 -> 35.21 | Hurt |
| Apply DPO on Mercury-train | deepseek-coder-6.7b-base · Mercury-eval | Beyond: 48.29 -> 49.70 | Helped |
Surprising findings
- SFT on the fastest solutions decreases Beyond scores for most models, while DPO improves efficiency.
- Efficiency gap (Beyond vs Pass) tends to widen for smaller models under DPO.
Limitations
- Assumes runtime is uniformly distributed, with limited solution samples per task (about 18.4 solutions).
- Possible data contamination of evaluated models with LeetCode data.
- Test case generators were initially produced by GPT-4 and manually revised, with coverage validated against LeetCode OJ.
Experimental setup
Datasets: Mercury-train Mercury-eval HumanEval MBPP
Benchmarks: Mercury-eval HumanEval MBPP
Metrics: Pass Beyond Gap runtime percentile
Baselines: SFT HumanEval MBPP
Builds on: DPO LoRA HumanEval MBPP LeetCode
Closest papers by meaning
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.91)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- StackEval: Benchmarking LLMs in Coding Assistance (NeurIPS 2024 · similarity 0.88)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.