Research map / Large language models / Instruction and fine-tuning
EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EffiInstruct, a 65k-task instruction dataset where multiple LLM-generated solutions are executed locally and the fastest/least-memory one is kept, then fine-tunes code LLMs (EffiCoder) to improve both correctness and efficiency.
Problem
LLM-generated code is correct but often much less efficient in execution time and memory than canonical human solutions, and existing code instruction-tuning datasets focus only on correctness.
Approach
Generate multiple candidate solutions per task with several LLMs, profile each by local execution on generated test cases, keep the most efficient (lowest time and memory) as the SFT target, and fine-tune open code LLMs on this efficiency-selected data.
Key results
- Qwen2.5-Coder-7B-Instruct pass@1 on EffiBench rises from 44.8% to 57.7% and average execution time for correct code drops 48.4% (0.31s to 0.16s).
- On HumanEvalPlus, CodeLlama-7b-hf fine-tuned with EffiInstruct reaches 31.1% pass@1 versus 19.5% for PIE and 9.1% for Mercury, with a 46.2% ET reduction.
- Fine-tuned DeepSeek-Coder-6.7B-instruct on HumanEvalPlus improves pass@1 from 47.6% to 72.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-Coder-7B-Instruct on EffiInstruct | Qwen2.5-Coder-7B-Instruct · EffiBench | pass@1 (%): 44.8 -> 57.7 | Helped |
| Fine-tune Qwen2.5-Coder-7B-Instruct on EffiInstruct | Qwen2.5-Coder-7B-Instruct · EffiBench | ET (s): 0.31 to 0.16; 48.4% | Helped |
| Fine-tune Qwen2.5-Coder-7B-Instruct on EffiInstruct | Qwen2.5-Coder-7B-Instruct · HumanEvalPlus | pass@1 (%): 76.2 -> 78.0 | Helped |
| Fine-tune Qwen2.5-Coder-7B-Instruct on EffiInstruct | Qwen2.5-Coder-7B-Instruct · EvoEval_creative | pass@1 (%): 44.0 -> 44.0 | No clear effect |
| Fine-tune Qwen2.5-Coder-7B-Instruct on EffiInstruct | Qwen2.5-Coder-7B-Instruct · DS-1000 | pass@1 (%): 17.00% to 35.70% | Helped |
| Fine-tune DeepSeek-Coder-6.7B-Instruct on EffiInstruct | DeepSeek-Coder-6.7B-Instruct · DS-1000 | pass@1 (%): 37.50% | No clear effect |
| Fine-tune CodeLlama-7b-hf on EffiInstruct rather than PIE (All) | CodeLlama-7b-hf · HumanEvalPlus | pass@1: 19.5% to 31.1% | Helped |
| Fine-tune CodeLlama-7b-hf on EffiInstruct rather than Mercury | CodeLlama-7b-hf · HumanEvalPlus | pass@1: 9.1% to 31.1% | Helped |
| Use PIE HQ instead of the CodeLlama-7b-hf baseline | CodeLlama-7b-hf · EffiBench | ET: 1.02 -> 1.14 | Hurt |
Surprising findings
- Gains on DS-1000 efficiency are modest (under 1% to about 5% ET reduction) even though pass@1 for Qwen2.5-Coder-7B-Instruct rises from 17.00% to 35.70%.
- Efficiency-selection data for CodeLlama on EffiBench gives a 16.7% ET reduction, while memory usage changes are negligible for several Qwen models (near 0%).
Limitations
- Efficiency measurements depend on local hardware and software environments; results are shown across several setups in the appendix but may vary.
- Test cases for some tasks are generated by GPT-3.5-turbo and filtered only by passing the initial solution, which may leave test coverage gaps.
- Improvements in memory usage are inconsistent across models and benchmarks, with some near zero.
Experimental setup
Datasets: SelfCodeAlign CodeFeedback-Filtered-Instruction Tested-143k-Python-Alpaca Glaive-Code-Assistant Magicoder-Evol-Instruct-110K Dolphin-Coder Magicoder-OSS-Instruct-75K Self-OSS-Instruct-SC2-Exec-Filter-50K APPS
Benchmarks: EffiBench HumanEvalPlus MBPPPlus EvoEval DS-1000 HumanEval-X ENAMEL
Metrics: pass@1 execution time (ET) normalized execution time (NET) max memory usage (MU) total memory usage (TMU) normalized TMU (NTMU) eff@1
Baselines: PIE Mercury EffiLearner Code Alpaca WizardCoder OSS-Instruct
Builds on: Self-Instruct Evol-Instruct OSS-Instruct SelfCodeAlign PIE
Closest papers by meaning
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.91)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.91)
- AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data (NeurIPS 2024 · similarity 0.90)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.89)
- LLM-Assisted Code Cleaning For Training Accurate Code Generators (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.