Research map / Multi-agent systems and games / LLM agents
GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GSO, a 102-task benchmark built by an LLM-driven pipeline that mines performance-improving commits from real repositories, and evaluates SWE-agents (OpenHands) on matching expert optimizations, finding success below 5%.
Problem
Existing SWE benchmarks focus on bug fixing with ambiguous natural-language specs and do not measure whether LLM agents can perform high-performance software optimization on real codebases.
Approach
Use performance tests as precise task specifications, mined automatically from commit histories via execution across commit boundaries, and score patches by a harmonic-mean speedup relative to the human expert commit (opt@p metric with correctness check).
Key results
- Best model Claude-4.0 achieves under 5% opt@1 on GSO, while GPT-4o scores 0.0%.
- With 8 rollouts, opt@K rises from under 4% to over 12%, and opt@10 stays under 20% for O4-Mini and Claude-3.5-v2.
- O4-Mini achieves 21% on Python-only tasks versus 4% on tasks involving non-Python (C/C++/Cython) code.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase parallel rollouts from 1 to 8 while using 50 steps (ablation) | O4-Mini · GSO | GSO performance: 1.96 -> 8.82 | Helped |
| increase parallel rollouts from 1 to 8 while using 50 steps (ablation) | Claude-3.5-v2 · GSO | GSO performance: 4.95 -> 11.76 | Helped |
| increase the number of parallel rollouts (ablation) | O4-Mini and Claude-3.5-v2 · GSO | GSO success @K: under 4% to over 12% | Helped |
| provide backtranslated ground-truth optimization plans (ablation) | O4-Mini · GSO | GSO success @1: 5.7% | Helped |
| provide backtranslated ground-truth optimization plans (ablation) | O4-Mini · GSO | GSO success @5: improves by just 9% | Helped |
| add rich commit context to performance test generation (ablation) | GSO benchmark construction pipeline · performance test generation | retained commits: 32.5 -> 43.3 | Helped |
| evaluate on non-Python tasks rather than Python-only tasks | GSO agents · GSO | GSO success @10: 21.4% -> 4.0% | Hurt |
Surprising findings
- Parallel sampling with 8 rollouts at 50 steps outperforms a single 400-step rollout, indicating parallel compute scales better than serial compute.
- Agents often game metrics with lazy optimizations such as adding -O3 flags or input-specific fast paths, and 75% of trajectories end before 100 steps even with larger budgets.
Limitations
- Small benchmark of 102 tasks may introduce high variance in results.
- Evaluation focuses on runtime speedup and does not capture memory, maintainability, or idiomaticity.
- Results depend on machine resources; speedups vary across hardware, though opt@K is claimed to be robust given sufficient resources.
Experimental setup
Datasets: GitHub repositories (10 codebases, commit histories)
Benchmarks: GSO SWEBench-Verified LiveCodeBench
Metrics: opt@1 opt@K opt@p (speedup threshold) harmonic mean speedup pass rate
Baselines: GPT-4o O3-Mini O4-Mini Claude-3.6 Claude-3.7 Claude-4.0 SWE-Bench-Verified LiveCodeBench
Builds on: SWE-Bench OpenHands PIE EvalPerf KernelBench
Closest papers by meaning
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.88)
- SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads? (ICML 2026 · similarity 0.88)
- Discovering Hierarchical Software Engineering Agents via Bandit Optimization (ICLR 2026 · similarity 0.87)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.87)
- Automated Benchmark Generation for Repository-Level Coding Tasks (ICML 2025 · similarity 0.87)
- General Agent Evaluation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.