AI Paper Atlas

Research map / Large language models / Agents, code and math

FormulaCode: Evaluating Agentic Optimization on Large Codebases

Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, Yisong Yue

ICML 2026 · Large language models › Agents, code and math · approach: Software engineering agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
LLM agents optimize the codebase instead of expert patchall agent-model configurations · FormulaCode-Vspeedup vs baseline: speedups 1.0346 to 1.0987 (>1)Improved
agent edits versus expert editsall agent-model configurations · FormulaCode-Vadvantage: best -0.0112 (OpenHands + Claude 4.0 Sonnet), expert speedup 1.1040Hurt
use OpenHands instead of Terminus 2 (ablation)GPT-5 · FormulaCode-Vadvantage: -0.0504 -> -0.0209Improved
use OpenHands instead of Terminus 2 (ablation)Claude 4.0 Sonnet · FormulaCode-Vadvantage: -0.0410 -> -0.0112Improved
use OpenHands instead of Terminus 2 (ablation)Qwen 3 Coder · FormulaCode-Vadvantage: -0.0454 -> -0.0301Improved
aggregate at function level vs module level (ablation)most agent configurations · FormulaCode-Vstratified advantageImproved
expert solution uses parallelization or batching (ablation)all agents · FormulaCode-Vadvantage vs expert: e.g. Parallel 0.010 to 0.015 for GPT-5Improved
expert solution delegates to lower-level implementations (C extensions, vectorization) (ablation)all agents · FormulaCode-Vadvantage vs expert: Lower: -0.233 to -0.720Hurt
most popular repositories (Q4, 6,371-10,343 stars) (ablation)agents · FormulaCode-Vadvantage: Terminus 2 + Claude 4.0 Sonnet -0.3529Hurt
mid-popularity repositories (20th-60th percentile) (ablation)agents · FormulaCode-Vadvantage: e.g. Terminus 2 + GPT-5 0.0423 in Q2Improved
use higher-priced frontier models instead of open weights (ablation)agent cost-effectiveness · FormulaCode-Vcost-weighted advantageImproved
task creation after model knowledge cutoff (ablation)Claude 4.0 Sonnet, GPT-5, Gemini 2.5 Pro · FormulaCode-Vmean global speedup: GPT-5 0-3 mo: 0.9871 before -> 1.0378 afterNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy