Research map / Large language models / Agents, code and math
AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AlgoTune, a 154-task benchmark scoring LMs by speedup of generated numerical code over reference library solvers, plus the AlgoTuner budgeted agent loop evaluated on four frontier models.
Problem
Existing code benchmarks check binary correctness on human-solved tasks; there is no open-ended measure of whether LMs can make numerical programs faster than reference implementations.
Approach
Score LM-written solvers by runtime speedup over reference solvers from popular Python libraries (NumPy, SciPy, CVXPY, sklearn, NetworkX), gated by a task-specific solution verifier, and evaluate with a budgeted SWE-Agent-style loop (AlgoTuner) that edits, profiles, and verifies code.
Key results
- o4-mini-high AlgoTuner achieves an AlgoTune score of 1.72x, with 59.7% of tasks sped up by at least 1.1x.
- R1 scores 1.70x, Gemini 2.5 Pro 1.51x, and Claude Opus 4 1.33x under a $1 per-task budget.
- Speedups are mostly surface-level (Numba rewrites, library swaps); no novel algorithmic improvements were found across the tested LMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AlgoTuner with o4-mini-high to optimize reference solvers | AlgoTune reference solvers · AlgoTune (154 tasks) | harmonic-mean speedup: 1.00× -> $1.72\times$ | Helped |
| Use AlgoTuner with DeepSeek R1 | AlgoTune reference solvers · AlgoTune (154 tasks) | harmonic-mean speedup: 1.00× -> $1.70\times$ | Helped |
| Use AlgoTuner with Gemini 2.5 Pro | AlgoTune reference solvers · AlgoTune (154 tasks) | harmonic-mean speedup: 1.00× -> $1.51\times$ | Helped |
| Use AlgoTuner with Claude Opus 4 | AlgoTune reference solvers · AlgoTune (154 tasks) | harmonic-mean speedup: 1.00× -> $1.33\times$ | Helped |
| Rewrite the CVXPY feedback-controller solver to use SciPy's discrete algebraic Riccati solver | feedback_controller_design · feedback_controller_design | speedup: 81 | Helped |
| Use symmetric eigendecomposition and apply eigenvalues directly to eigenvectors | psd_cone_projection · psd_cone_projection | speedup: 8 | Helped |
| Replace NetworkX objects with adjacency lists and a single Weisfeiler–Lehman pass | graph_isomorphism · graph_isomorphism | speedup: 52x | Helped |
| Use BLAS operations instead of pure Python | communicability solver · communicability | speedup: more than 142x | Helped |
| Replace the SciPy solver with a Numba kernel | ode_hodgkinhuxley · ode_hodgkinhuxley | speedup: 112x | Helped |
| Compile a Numba kernel that streams once over the data | Wasserstein distance solver · wasserstein_dist | speedup: more than 4x | Helped |
| Use the same implementation as the reference | svm solver · svm | speedup: 1.00 | No clear effect |
| Replace scikit-learn's optimized Lasso with a pure-Python and Numba regressor | lasso solver · lasso | runtime relative to reference: 0.33x | Hurt |
Surprising findings
- Models find surface-level optimizations rather than algorithmic innovations.
- o4-mini and R1 reach better scores after spending $0.10 than Claude Opus 4 and Gemini 2.5 Pro reach after the full $1 budget.
Limitations
- Verification relies on fixed test inputs and verifier completeness, so loophole solutions may pass.
- Each agent/LM/task combination was run only once due to budget, with no error bars.
- Benchmark is limited to tasks with manually written generators and verifiers.
Experimental setup
Datasets: AlgoTune tasks (154 tasks, 5-task dev set)
Benchmarks: AlgoTune
Metrics: speedup over reference (harmonic mean) percentage of tasks with at least 1.1x speedup
Baselines: o4-mini-high Claude Opus 4 Gemini 2.5 Pro DeepSeek R1 0528
Builds on: SWE-Agent LiteLLM Cython Numba
Closest papers by meaning
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.88)
- CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.88)
- SciCode: A Research Coding Benchmark Curated by Scientists (NeurIPS 2024 · similarity 0.87)
- LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? (NeurIPS 2025 · similarity 0.87)
- AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Tasks (NeurIPS 2025 · similarity 0.87)
- Algorithmic progress in language models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.