Research map / Large language models / AI and society
Measuring AI Ability to Complete Long Software Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the 50%-task-completion time horizon metric, measuring the human task length that frontier AI agents complete with 50% success across HCAST, RE-Bench, and a new 66-task SWAA suite, and finds the horizon doubling roughly every seven months since 2019.
Problem
Benchmark scores are hard to interpret in real-world terms and saturate quickly, and there is no general, intuitive quantitative metric for comparing AI capabilities across models and benchmarks of very different difficulty.
Approach
Measure AI capability as the human time-to-complete of tasks an agent solves with 50% success, by fitting a logistic model of agent success against log human baseline time (with task difficulty set by skilled-human baseline times rather than learned ratings), and track this horizon against model release dates.
Key results
- The 50% time horizon of o3 is around 110 minutes on the combined task suite.
- The 50% time horizon has doubled approximately every 207 days (95% CI 166-240 days) from 2019 to 2025.
- The 80% time horizon doubles in about 204 days, with horizons roughly 4-6x shorter than the 50% horizon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use newer frontier models rather than earlier models | AI agents on HCAST + SWAA + RE-Bench · HCAST + SWAA + RE-Bench | 50% time horizon: 2 seconds -> 110-minute | Helped |
| Use the 80% success-rate horizon instead of the 50% success-rate horizon | Frontier models · HCAST + SWAA + RE-Bench | time horizon: 4-6x shorter | Hurt |
| Evaluate on SWE-bench Verified using its task-time annotations | Frontier models · SWE-bench Verified | time horizon doubling time: 143 days -> around 70 days | Helped |
| Increase task messiness | Models evaluated on HCAST and RE-Bench · HCAST + RE-Bench | performance | Hurt |
| Use contractor rather than repository-maintainer completion time as task length | AI agents on internal PR tasks · Internal PRs | alignment with predicted performance | Helped |
Surprising findings
- Human baseliners' own time horizon is only around 1.5 hours, far below the 8 hours they were paid for, which the authors attribute to incentive-driven early quitting.
- SWE-bench Verified shows a shorter doubling time (around 70 days) than the main suite, likely due to annotator time estimates underestimating contractor times on easy tasks.
Limitations
- Tasks are automatically scored, short, low-context and lack real-world messiness, so external validity is uncertain.
- Human baselines are conditioned on success, biasing task length estimates, especially for long tasks with success rates below 50%.
- Few frontier models in the 2024-2025 window make the recent trend estimate low confidence.
Experimental setup
Datasets: HCAST RE-Bench SWAA SWE-bench Verified
Benchmarks: HCAST RE-Bench SWAA SWE-bench Verified
Metrics: 50% time horizon 80% time horizon task success rate doubling time
Baselines: GPT-4 1106 o1 Claude 3.5 Sonnet GPT-4o
Builds on: item response theory HCAST RE-Bench SWE-bench Verified
Closest papers by meaning
- RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts (ICML 2025 · similarity 0.87)
- SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work? (NeurIPS 2026 · similarity 0.84)
- AGI-Elo: How Far Are We From Mastering A Task? (NeurIPS 2025 · similarity 0.84)
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (ICLR 2026 · similarity 0.83)
- UltraHorizon: Benchmarking LLM-Agent Capabilities in Ultra Long-Horizon Scenarios (ICML 2026 · similarity 0.83)
- Offloading Score: Measuring AI Reliance through Counterfactual Workflows (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.