Research map / Large language models / Alignment and preferences
Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Afterburner (LLM code optimizer) and Monolith (execution sandbox) in a closed-loop iterative framework, comparing SFT, DPO and GRPO for improving code efficiency, with GRPO giving sustained gains on the new Venus benchmark and APPS.
Problem
LLM-generated code is functionally correct but often inefficient in time and memory; existing approaches (prompting, SFT, correctness-focused RL) do not reliably improve efficiency at test time.
Approach
A closed-loop test-time iterative optimization where an LLM refines code using empirical time/memory feedback from a sandbox, with the model trained via GRPO using a composite format/correctness/efficiency reward; percentile ranks against human solutions measure efficiency.
Key results
- Afterburner-GRPO (Qwen2.5-3B) raises Pass@1 from 47% to 62% and Beyond-I from 31% to 45% on Venus, versus 48.67 Pass@1 and 22.50 Beyond-I for Afterburner-SFT at iteration 10.
- OpenAI o4-mini reaches 89.11% Pass@1 and 56.85% Beyond-T on Venus, while the 3B GRPO model reaches 61.67% Pass@1 and 45.17% Beyond-T.
- Removing execution feedback drops Afterburner-GRPO Beyond-T from 40.81 to 34.15 at iteration 4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GRPO instead of SFT | Afterburner on Qwen 2.5 3B · Venus | Beyond-T: 26.78 -> 45.17 | Helped |
| use GRPO instead of SFT | Afterburner on Qwen 2.5 3B · Venus | Pass@1: 48.67 -> 61.67 | Helped |
| add execution feedback (ablation) | Afterburner-GRPO · Venus iteration 4 | Beyond-T: 34.15 -> 40.81 | Helped |
| add execution feedback (ablation) | Afterburner-GRPO · Venus iteration 4 | Pass@1: 52.51 -> 57.00 | Helped |
| add original code input (ablation) | Afterburner-GRPO · Venus iteration 4 | Beyond-I: 24.24 -> 33.51 | Helped |
| add original code input (ablation) | Afterburner-GRPO · Venus iteration 4 | Beyond-T: 32.17 -> 40.81 | Helped |
| add execution feedback (ablation) | Afterburner-SFT · Venus iteration 4 | Beyond-M: 24.70 -> 24.39 | Hurt |
| add original code input (ablation) | Afterburner-SFT · Venus iteration 4 | Beyond-M: 26.17 -> 24.39 | Hurt |
| add execution feedback (ablation) | Afterburner-DPO · Venus iteration 4 | Beyond-I: 25.68 -> 27.89 | Helped |
| add original code input (ablation) | Afterburner-DPO · Venus iteration 4 | Beyond-I: 22.01 -> 27.89 | Helped |
| use GRPO online execution-feedback training | Afterburner · Venus, iterative optimization | Beyond-I: 31% to 45% | Helped |
| use GRPO online execution-feedback training | Afterburner · Venus, iterative optimization | Pass@1: 47% to 62% | Helped |
Surprising findings
- Afterburner-GRPO produces the highest share of solutions better than all human solutions (8.00% time), exceeding o4-mini's 7.00%, though it also yields more worse-than-human solutions.
Limitations
- Scope limited to competition-level algorithmic problems; not tested on real-world software projects or non-algorithmic efficiency (I/O, libraries).
- Iterative test-time optimization requires substantially more inference compute than single-pass generation.
- Efficiency measurements are noisy and hardware-dependent, and the reward design relies on sandbox measurements.
Experimental setup
Datasets: Venus APPS
Benchmarks: Venus APPS
Metrics: Pass@1 Beyond-T Beyond-M Beyond-I B%/M%/W%/F%
Baselines: SFT DPO Qwen 2.5 Coder 7B OpenAI o4-mini QwQ 32B DeepSeek V3 Claude 3.7 Sonnet GPT-4o
Builds on: GRPO DPO Mercury EffiBench CodeRL
Closest papers by meaning
- RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025 · similarity 0.90)
- On-Policy Optimization with Group Equivalent Preference for Multi-Programming Language Understanding (NeurIPS 2025 · similarity 0.89)
- CodeScaler: Scaling Code LLM Training and Test-Time Inference via Reward Models (NeurIPS 2026 · similarity 0.89)
- Reinforcement Learning for Code Optimization (NeurIPS 2026 · similarity 0.89)
- MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation (NeurIPS 2026 · similarity 0.88)
- CURE: Co-Evolving Coders and Unit Testers via Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.