Research map / Large language models / Agents, code and math
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SWE-Lancer, a benchmark of 1,488 real Upwork freelance software engineering tasks (IC SWE with end-to-end Playwright tests and SWE Manager proposal selection) worth $1M, and evaluates GPT-4o, o1, and Claude 3.5 Sonnet.
Problem
Existing coding benchmarks rely on unit tests and open-source repos, and do not measure full-stack engineering, managerial judgment, or economic value of AI software engineering work.
Approach
Source real freelance tasks and their actual payouts from Upwork (Expensify repo), grade IC tasks with professionally written end-to-end browser tests, and grade manager tasks by matching the originally chosen proposal, mapping model performance to dollars earned.
Key results
- Claude 3.5 Sonnet is the top model, scoring 26.2% pass@1 on Diamond IC SWE tasks and 44.9% on Diamond SWE Manager tasks, earning $208K of $500.8K on the Diamond set.
- On the full 1,488-task set, the best model (Claude 3.5 Sonnet) earns about $403K of $1M (33.7% pass@1).
- o1 pass@1 on Diamond IC SWE rises from 9.3% (low reasoning effort) to 16.5% (high effort); with 5 attempts, o1 yields 33.5% cost savings over freelancer-only baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase reasoning effort from low to high (ablation) | o1 · IC SWE (Diamond) | pass@1: 9.3% -> 16.5% | Helped |
| increase reasoning effort from low to high (ablation) | o1 · IC SWE (Diamond) | earnings: $16K to $29K | Helped |
| remove user tool (ablation) | o1 · IC SWE (Diamond) | pass@1: 16.5% -> 13.1% | Hurt |
| remove user tool (ablation) | GPT-4o · IC SWE (Diamond) | pass@1: 8.0% -> 8.0% | No clear effect |
| allow more attempts (pass@k rather than pass@1) (ablation) | GPT-4o and o1 · IC SWE (Diamond) | pass rate | Helped |
| allow six additional attempts (ablation) | o1 · IC SWE (Diamond) | pass rate: nearly triples | Helped |
| allow six attempts rather than one (ablation) | GPT-4o · IC SWE (Diamond) | pass rate: 16.5% | Helped |
| use Claude 3.5 Sonnet rather than o1 | IC SWE (Diamond) · IC SWE (Diamond) | pass@1: 9.7% | Helped |
| use Claude 3.5 Sonnet rather than o1 | SWE Manager (Diamond) · SWE Manager (Diamond) | pass@1: 3.4% | Helped |
Surprising findings
- Models perform better on SWE Manager tasks than on IC SWE tasks, with manager pass@1 often more than double IC pass rates.
- GPT-4o with pass@6 matches o1 with pass@1, and o1 pass@k roughly triples with 6 extra attempts.
Limitations
- Tasks drawn only from the Expensify repository and Upwork, limiting diversity; infrastructure tasks underrepresented.
- Contamination risk since tasks are public GitHub issues from 2023-2024; the full set is private and available only on request.
- Text-only evaluation; models cannot view screenshots or videos, and cannot ask clarifying questions.
Experimental setup
Datasets: SWE-Lancer (Upwork Expensify tasks)
Benchmarks: SWE-Lancer SWE-Lancer Diamond
Metrics: pass@1 pass@k earnings (payout dollars) earn rate cost savings
Baselines: GPT-4o o1 Claude 3.5 Sonnet DeepSeek-R1 Llama 3.3
Builds on: SWE-bench SWE-bench Verified Playwright
Closest papers by meaning
- SWE-smith: Scaling Data for Software Engineering Agents (NeurIPS 2025 · similarity 0.83)
- SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents (NeurIPS 2025 · similarity 0.82)
- SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.82)
- SWE-bench Goes Live! (NeurIPS 2025 · similarity 0.81)
- SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement (ICLR 2025 · similarity 0.81)
- BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows (NeurIPS 2026 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.