Research map / Large language models / Agents, code and math
FormulaCode: Evaluating Agentic Optimization on Large Codebases
Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, Yisong Yue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| LLM agents optimize the codebase instead of expert patch | all agent-model configurations · FormulaCode-V | speedup vs baseline: speedups 1.0346 to 1.0987 (>1) | Improved |
| agent edits versus expert edits | all agent-model configurations · FormulaCode-V | advantage: best -0.0112 (OpenHands + Claude 4.0 Sonnet), expert speedup 1.1040 | Hurt |
| use OpenHands instead of Terminus 2 (ablation) | GPT-5 · FormulaCode-V | advantage: -0.0504 -> -0.0209 | Improved |
| use OpenHands instead of Terminus 2 (ablation) | Claude 4.0 Sonnet · FormulaCode-V | advantage: -0.0410 -> -0.0112 | Improved |
| use OpenHands instead of Terminus 2 (ablation) | Qwen 3 Coder · FormulaCode-V | advantage: -0.0454 -> -0.0301 | Improved |
| aggregate at function level vs module level (ablation) | most agent configurations · FormulaCode-V | stratified advantage | Improved |
| expert solution uses parallelization or batching (ablation) | all agents · FormulaCode-V | advantage vs expert: e.g. Parallel 0.010 to 0.015 for GPT-5 | Improved |
| expert solution delegates to lower-level implementations (C extensions, vectorization) (ablation) | all agents · FormulaCode-V | advantage vs expert: Lower: -0.233 to -0.720 | Hurt |
| most popular repositories (Q4, 6,371-10,343 stars) (ablation) | agents · FormulaCode-V | advantage: Terminus 2 + Claude 4.0 Sonnet -0.3529 | Hurt |
| mid-popularity repositories (20th-60th percentile) (ablation) | agents · FormulaCode-V | advantage: e.g. Terminus 2 + GPT-5 0.0423 in Q2 | Improved |
| use higher-priced frontier models instead of open weights (ablation) | agent cost-effectiveness · FormulaCode-V | cost-weighted advantage | Improved |
| task creation after model knowledge cutoff (ablation) | Claude 4.0 Sonnet, GPT-5, Gemini 2.5 Pro · FormulaCode-V | mean global speedup: GPT-5 0-3 mo: 0.9871 before -> 1.0378 after | No clear effect |
Closest papers by meaning
- CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments (ICML 2026 · similarity 0.89)
- SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks (NeurIPS 2026 · similarity 0.89)
- Steering Large Language Models between Code Execution and Textual Reasoning (ICLR 2025 · similarity 0.88)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.88)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (NeurIPS 2026 · similarity 0.88)
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
