Research map / Large language models / Agents, code and math
SWE-Pro: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization
Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-Pro, a benchmark for evaluating LLMs on performance-oriented optimization across real-world repositories, going beyond isolated functions and single metrics.
Problem
Existing benchmarks for LLM-based code optimization oversimplify the task by focusing on isolated functions or a single performance metric, missing trade-offs between execution time and other resources in real-world codebases.
Approach
A repository-level benchmark for performance optimization that captures how optimization happens in real codebases and considers multiple performance dimensions and their trade-offs, rather than isolated functions or a single metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from single-shot to agentic repository exploration (ablation) | LLMs (GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Flash) · SWE-Opt | correctness rate (Test@1): max 79.4% -> up to 94.1% | Improved |
| move from single-shot to agentic repository exploration (ablation) | LLMs · SWE-Opt | runtime performance gain: runtime pass rates 14.7%-17.6% | No clear effect |
| move from single-shot to agentic repository exploration (ablation) | Claude Sonnet 4.6 · SWE-Opt | runtime IF: 6.67x -> 25.29x | Improved |
| use BM25 retrieval instead of oracle context (ablation) | LLMs · SWE-Opt | patch and correctness rates | Hurt |
| use BM25 retrieval instead of oracle context (ablation) | Claude Sonnet 4.6 · SWE-Opt | runtime pass rate: 2.0% -> 12.7% | Improved |
| LLM-generated patches versus expert patches | GPT-5.2 · SWE-Opt Oracle | runtime IF: 0.69x | Hurt |
| LLM patches on memory optimization | all single-shot LLMs · SWE-Opt | peak memory and TWMU | No clear effect |
| use expert-written patches instead of LLM patches | Gold solution · SWE-Opt | runtime pass rate: 93.1% vs max 12.7% for LLMs | Improved |
| increase workload input size | PR 7578 Claude patch · SWE-Opt | runtime IF: around 100x on large workloads | Improved |
Experimental setup
Benchmarks: SWE-Pro
Metrics: execution time resource trade-offs
Closest papers by meaning
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.92)
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code (NeurIPS 2025 · similarity 0.89)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.89)
- GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents (NeurIPS 2025 · similarity 0.88)
- Mercury: A Code Efficiency Benchmark for Code Large Language Models (NeurIPS 2024 · similarity 0.88)
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
