Research map / Large language models / AI and society
Working with AI: Measuring the Applicability of Generative AI to Occupations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measures AI applicability to occupations by classifying 200k Bing Copilot conversations into O*NET intermediate work activities (user goals and AI actions), scoring completion and scope to derive occupation-level AI applicability.
Problem
Existing estimates of occupational AI impact rely on predicted AI capabilities or firm-level data; the paper asks which work activities and occupations real-world generative AI usage actually covers and succeeds at.
Approach
Use real-world Copilot conversations, classified exhaustively into O*NET intermediate work activities on both the user-goal and AI-action side, combined with LLM-judged completion and scope, to compute an occupation-level AI applicability score that separates delegation from assistance.
Key results
- Employment-weighted occupation-level correlation between AI applicability score and Eloundou et al. E1 predictions is r = 0.73.
- Completion is highly correlated with thumbs feedback (weighted r > 0.75), and scope correlates with log IWA share at r = 0.64.
- AI applicability has only a weak employment-weighted correlation with wage (r = 0.13).
Surprising findings
- AI applicability is only weakly correlated with wage, contrary to prior work finding strong positive correlations.
- Most occupations show at least some AI applicability, with information work dominating the top scores.
Limitations
- Conversation data cannot be released; only aggregated IWA and occupation metrics are public, and only one platform (consumer Copilot, US) is represented.
- Human-LLM agreement on IWA matching and scope classification is low to moderate (kappa roughly 0.3-0.5).
- Usage-based measures do not capture productivity impact or employment and wage effects; O*NET is US-centric and may lag.
Experimental setup
Datasets: Microsoft Bing Copilot conversations (US, Jan-Sep 2024, ~100k uniform sample plus ~100k thumbs-rated sample) O*NET 29.0 BLS OEWS 2023 CPS 2024
Benchmarks: Copilot thumbs feedback Eloundou et al. E1 exposure comparison BLS OEWS wage and employment
Metrics: AI applicability score completion rate scope share Cohen's kappa r_wg Pearson correlation
Baselines: Eloundou et al. E1 exposure
Builds on: O*NET Eloundou et al. GPT exposure measure Handa et al. Claude occupational usage analysis
Closest papers by meaning
- A Mathematical Framework for AI-Human Integration in Work (ICML 2025 · similarity 0.86)
- JobBench: Aligning Agent Work With Human Will (NeurIPS 2026 · similarity 0.85)
- Offloading Score: Measuring AI Reliance through Counterfactual Workflows (NeurIPS 2026 · similarity 0.85)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (NeurIPS 2025 · similarity 0.83)
- When AI Agents Compete for Jobs: Strategic Capabilities and Economic Dynamics of AI Labour Markets (ICML 2026 · similarity 0.83)
- Position: Societal Impacts Research Requires Benchmarks for Creative Composition Tasks (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.