Research map / Large language models / AI and society

Offloading Score: Measuring AI Reliance through Counterfactual Workflows

NeurIPS 2026 · Large language models › AI and society · approach: Social-science view of AI evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces offloading score, a simulation-based reliance metric that measures the fraction of workflow steps saved by AI tool use relative to LLM-generated human-only counterfactual workflows, and validates it in a 40-participant coding study under time pressure.

Problem

Existing reliance measures rely on binary output adoption or self-reports and fail to capture how task effort is distributed between user and AI tool in multi-turn human-AI workflows.

Approach

Replace each AI-assisted workflow step with an LLM-simulated human-only counterfactual sequence and score reliance as the fraction of counterfactual steps saved by the tool, plus descriptive process and output-use labels.

Key results

• Offloading score is higher in the short-time condition (0.451 vs 0.308, p=0.018), while baseline measures such as code attribution (p=0.072) and NASA-TLX load (p=0.881) do not significantly separate conditions.
• Over 85% of sampled counterfactual steps were rated plausible (>=3 on 5-point Likert scale) by human judges.
• Among non-outlier users, offloading score correlates with system recall at r=-0.440; overall correlation is -0.145.

Surprising findings

• Usage-based and self-reported baselines, including NASA-TLX load, fail to distinguish time-pressure conditions that the offloading score separates.
• A cluster of high-offloading, high-recall users appears to use the tool for learning, suggesting high reliance can be appropriate in some contexts.

Limitations

• Validated against only a single reference variable (time pressure) with 40 participants on programming tasks.
• Counterfactuals estimate an average user rather than personalized behavior and depend on imperfect workflow induction.
• Assumes cognitive effort can be approximated by externally observable workflow steps.

Experimental setup

Datasets: user study traces from 40 freelance developers prior human and agent software engineering workflow dataset

Benchmarks: controlled user study (short 1h vs long 4h time limits)

Metrics: offloading score t-test p-value Pearson correlation with condition system recall correlation Wilcoxon and permutation test on counterfactual similarity annotation agreement

Baselines: fraction of AI-attributed code number of AI interactions fraction of time using tool NASA-TLX cognitive load

Builds on: Flower and Hayes cognitive process model workflow induction toolkit

Closest papers by meaning

• Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies (NeurIPS 2026 · similarity 0.86)
• Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents (AAAI 2026 · similarity 0.85)
• Working with AI: Measuring the Applicability of Generative AI to Occupations (NeurIPS 2026 · similarity 0.85)
• A Mathematical Framework for AI-Human Integration in Work (ICML 2025 · similarity 0.84)
• JobBench: Aligning Agent Work With Human Will (NeurIPS 2026 · similarity 0.84)
• Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor (NeurIPS 2025 · similarity 0.84)