Research map / Large language models / Reasoning and chain of thought
EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EffiLearner, a training-free self-optimization loop where an LLM revises its generated code using line-level execution-time and memory profiles, improving efficiency of correct code on EffiBench, HumanEval, and MBPP.
Problem
LLM-generated code is often correct but inefficient in execution time and memory usage, and existing feedback methods focus mainly on correctness rather than efficiency.
Approach
Feed line-level execution-time and memory profiles (from line_profiler and memory_profiler) back to the same LLM over several iterations so it can revise code to reduce overhead, while only keeping code that passes open test cases.
Key results
- StarCoder2-15B on EffiBench: execution time reduced from 0.93 s to 0.12 s (87.1%) and TMU from 22.02 to 2.03 Mb*s (90.8%).
- GPT-3.5-Turbo-0301 on EffiBench: execution time reduced 22.2% (0.36 s to 0.28 s) and TMU reduced 92.1% (157.50 to 12.43 Mb*s).
- Unsupervised and result-aware self-refine baselines increased CodeLlama-70B TMU substantially (e.g., 203.92 to 1261.83 Mb*s for unsupervised), whereas EffiLearner reduced it to 14.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use overhead-profile feedback for iterative code self-optimization | StarCoder2-15B · EffiBench | ET (s): 0.93 -> 0.12; 87.1% | Helped |
| Use overhead-profile feedback for iterative code self-optimization | StarCoder2-15B · EffiBench | TMU (Mb*s): 22.02 -> 2.03; 90.8% | Helped |
| Use overhead-profile feedback for iterative code self-optimization | DeepSeek-6.7B-Ins · EffiBench | MU (Mb): 259.73 -> 36.97; 85.8% | Helped |
| Use overhead-profile feedback for iterative code self-optimization | GPT-3.5-Turbo-0301 · EffiBench | ET (s): 0.36 -> 0.28; 22.2% | Helped |
| Use overhead-profile feedback for iterative code self-optimization | CodeLlama-70B · EffiBench | ET (s): 0.52 -> 0.47; 9.6% | Helped |
| Use overhead-profile feedback for iterative code self-optimization | StarCoder2-15B · EffiBench | MU (Mb): 26.35 -> 27.67; -5.0% | Hurt |
| Replace detailed profiler feedback with no additional feedback (ablation) | CodeLlama-70B · EffiBench | ET (s): 0.52 -> 0.79; -51.9% | Hurt |
| Replace detailed profiler feedback with result metrics (ET, MU, TMU) (ablation) | CodeLlama-70B · EffiBench | TMU (Mb*s): 203.92 -> 1270.93; -523.2% | Hurt |
| Use memory-profiler feedback only instead of both profiles (ablation) | CodeLlama-70B · EffiBench | ET (s): 0.52 -> 0.53; -1.9% | Hurt |
| Use execution-time-profiler feedback only instead of both profiles (ablation) | CodeLlama-70B · EffiBench | ET (s): 0.52 -> 0.51; 1.9% | Hurt |
| Increase self-optimization from one to five steps (ablation) | GPT-3.5-Turbo-0301 · EffiBench | ET (s): 0.33 -> 0.28 | Helped |
| Increase self-optimization from one to five steps (ablation) | CodeLlama-70B · EffiBench | TMU (Mb*s): 14.91 -> 14.53 | Helped |
Surprising findings
- Generic self-refine prompts without profiler feedback often degraded efficiency, sometimes sharply increasing memory usage (e.g., CodeLlama-70B TMU up roughly 6x).
- Optimized code's pass@1 can drop slightly (up to about 0.5 points) because only open test cases guide optimization.
Limitations
- Iterative self-optimization is time-consuming and profile feedback consumes extra tokens.
- Evaluated only on Python.
- Optimized code may slightly reduce pass@1 because only open test cases guide optimization.
Experimental setup
Datasets: EffiBench HumanEval MBPP HumanEval-Plus MBPP-Plus
Benchmarks: EffiBench HumanEval MBPP
Metrics: Execution Time (ET) Normalized Execution Time (NET) Max Memory Usage (MU) Normalized Max Memory Usage (NMU) Total Memory Usage (TMU) Normalized Total Memory Usage (NTMU) pass@1
Baselines: Unsupervised Self-Refine Result-Aware Self-Refine Self-Edit PIE Critic Supersonic
Builds on: Self-Refine EffiBench EvalPlus Self-Debug
Closest papers by meaning
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.92)
- EffiBench: Benchmarking the Efficiency of Automatically Generated Code (NeurIPS 2024 · similarity 0.91)
- EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning (ICML 2025 · similarity 0.91)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.90)
- Teaching Large Language Models to Self-Debug (ICLR 2024 · similarity 0.88)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.