Research map / Multi-agent systems and games / Human-AI and goal-conditioned
JobBench: Aligning Agent Work With Human Will
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces JobBench, a 130-task benchmark of occupational agentic workflows grounded in worker-reported delegation preferences (Workbank), graded by chained binary rubrics, and evaluates 36 model-scaffold configurations, with the best reaching 45.9%.
Problem
Existing occupational agent benchmards are scoped by economic value and replacement framing, ignoring which duties workers actually want delegated to AI. Their scoring also gives credit for final deliverables without verifying source-grounded reasoning.
Approach
Select occupations and work duties using worker-reported automation desire from Workbank combined with economic exposure, build heterogeneous multi-file workspaces for each task, and grade with chained rubrics where a rubric earns credit only if every binary criterion in its chain passes.
Key results
- Claude Opus 4.7 under Claude Code scores 45.9% on JobBench-Main, the highest of 36 configurations; GPT-5.5 under Codex scores 42.7%.
- Outside Claude and GPT families no configuration exceeds 19%; Grok-4.2-Fast scores 4.38%.
- Grok-4.1-Fast judge tracks Opus 4.5 judge within 0.1-0.7 points while cutting full-run judge cost roughly 20x (about $40 to $2).
Surprising findings
- Attention from research papers and YC startups correlates negatively with agent capability on occupations (r=-0.15 papers, r=-0.34 YC).
- Scaffold choice shifts overall scores by several points for a fixed base model (e.g., Sonnet 4.6 36.9 under Claude Code vs 30.6 under OpenClaw).
Limitations
- Limited to U.S.-centered, digital, document-heavy occupational tasks across 35 O*NET occupations.
- Judge is an LLM; the paper notes ambiguity causes judge disagreement, and rubric validity depends on annotator design.
- Not intended for deciding worker replacement or deployment readiness.
Experimental setup
Datasets: Workbank OEWS 2024 arXiv agentic LLM abstracts YC company descriptions
Benchmarks: JobBench-Main JobBench-Easy
Metrics: normalized rubric score per-category average score runtime judge cost
Baselines: GDPVal Remote Labor Index $OneMillion-Bench PRBench
Builds on: Workbank GDPVal O*NET
Closest papers by meaning
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.87)
- When AI Agents Compete for Jobs: Strategic Capabilities and Economic Dynamics of AI Labour Markets (ICML 2026 · similarity 0.87)
- A Mathematical Framework for AI-Human Integration in Work (ICML 2025 · similarity 0.86)
- Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment (AAAI 2025 · similarity 0.86)
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.86)
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.