Research map / Large language models / Agents, code and math
\$OneMillion-Bench: How Far are Language Agents from Human Experts?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces $OneMillion-Bench, a 400-task expert-curated benchmark across five professional domains that scores LM agents with rubric-based expert scores, pass rates, and dollar-valued economic value.
Problem
Existing agent benchmarks are mostly exam-style or structured tasks and do not measure whether agents can perform economically consequential professional work.
Approach
Expert-curated professional tasks (200 English, 200 Chinese) with asymmetric negative rubrics, plus an economic-value metric that prices each task by expert time times market wage and credits an agent only for tasks it passes (expert score >= 0.7).
Key results
- Claude-Opus-4.6 leads vanilla models with 55.0% Expert Score and 36.5% Pass Rate on the Global set; with web search it reaches 63.0% and 43.5%.
- Web search helps strong models but can hurt weaker ones, e.g., Hunyuan-2.0 and Step-3.5-Flash decline after enabling search.
- Many agents reach 45-50% Expert Score but Pass Rate often stays below 25%, indicating partial rubric satisfaction rather than full task completion.
Surprising findings
- Claude-Opus-4.6 without web search outperforms its OpenRouter web-search setup, suggesting scaffold quality matters more than search access.
- Pass^k decays toward zero as k grows even though pass@k keeps rising, showing consistency collapse in repeated sampling.
Limitations
- Economic value depends on expert time estimates and wage anchoring, which are approximate.
- Web search results are sensitive to the scaffold used (official vs OpenRouter), limiting transferability of comparisons.
- Rubric scoring is case-by-case and hand-crafted, which may limit scale and generality.
Experimental setup
Datasets: $OneMillion-Bench
Benchmarks: $OneMillion-Bench
Metrics: Expert Score Pass Rate Economic Value pass@k pass^k
Baselines: GDPval xbench APEX-Agents RLI o3-DeepResearch o4-Mini-DeepResearch Sonar-DeepResearch
Builds on: GDPval xbench APEX-Agents RLI
Closest papers by meaning
- AgentBench: Evaluating LLMs as Agents (ICLR 2024 · similarity 0.91)
- LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth (ICML 2026 · similarity 0.90)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.90)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.90)
- MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? (NeurIPS 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.