AI Paper Atlas

Research map / Large language models / Agents, code and math

SWE-Pro: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization

Ezgi Sarıkayak, Wenchao Gu, Hesham Ghonim, Chunyang Chen

NeurIPS 2026 · Large language models › Agents, code and math · approach: Software engineering agents

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SWE-Pro, a benchmark for evaluating LLMs on performance-oriented optimization across real-world repositories, going beyond isolated functions and single metrics.

Problem

Existing benchmarks for LLM-based code optimization oversimplify the task by focusing on isolated functions or a single performance metric, missing trade-offs between execution time and other resources in real-world codebases.

Approach

A repository-level benchmark for performance optimization that captures how optimization happens in real codebases and considers multiple performance dimensions and their trade-offs, rather than isolated functions or a single metric.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
move from single-shot to agentic repository exploration (ablation)LLMs (GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Flash) · SWE-Optcorrectness rate (Test@1): max 79.4% -> up to 94.1%Improved
move from single-shot to agentic repository exploration (ablation)LLMs · SWE-Optruntime performance gain: runtime pass rates 14.7%-17.6%No clear effect
move from single-shot to agentic repository exploration (ablation)Claude Sonnet 4.6 · SWE-Optruntime IF: 6.67x -> 25.29xImproved
use BM25 retrieval instead of oracle context (ablation)LLMs · SWE-Optpatch and correctness ratesHurt
use BM25 retrieval instead of oracle context (ablation)Claude Sonnet 4.6 · SWE-Optruntime pass rate: 2.0% -> 12.7%Improved
LLM-generated patches versus expert patchesGPT-5.2 · SWE-Opt Oracleruntime IF: 0.69xHurt
LLM patches on memory optimizationall single-shot LLMs · SWE-Optpeak memory and TWMUNo clear effect
use expert-written patches instead of LLM patchesGold solution · SWE-Optruntime pass rate: 93.1% vs max 12.7% for LLMsImproved
increase workload input sizePR 7578 Claude patch · SWE-Optruntime IF: around 100x on large workloadsImproved

Experimental setup

Benchmarks: SWE-Pro

Metrics: execution time resource trade-offs

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy