Research map / Large language models / AI and society
Balancing Humans and Machines: A Study on Integration Scale and Its Impact on Collaborative Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a statistical framework decomposing ensemble accuracy into individual accuracy and diversity to study how the human-to-machine ratio and ensemble size affect collaborative performance, validated on CIFAR10H and ImageNet-16H.
Problem
Large-scale collaboration between multiple humans and multiple machines is under-studied; it is unclear how ensemble accuracy depends on ensemble size and the human-machine mix, and where resource-efficient returns diminish.
Approach
Extends ensemble error decomposition to multi-class classification with Gaussian and conditional-independence assumptions, yielding ensemble accuracy as a product of individual accuracy (Δ/σ) and a diversity term V; this predicts a performance ceiling, an optimal human-machine ratio trajectory, and a knee point as ensemble size grows.
Key results
- On ImageNet-16H, the ACC2 estimate tracks actual ensemble accuracy closely, with average absolute error 0.032 versus 0.052 for ACC1 and 0.624 for ACC3 (ACC3 only applicable at M≤2).
- On CIFAR10-H, ACC2 average absolute error is 0.172 versus 0.175 for ACC1 and 0.259 for ACC3.
- For a specific task with ensemble size five, the optimal ratio is two humans to three machines; on ImageNet-16H estimated knees align with actual knees, while on CIFAR10H the actual knee is at M=3 and estimated at M=4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACC2 to estimate ensemble accuracy | Human-machine ensembles · ImageNet-16H and CIFAR10H | absolute estimation error: 0.032; 0.172 | Helped |
| Use ACC2 instead of ACC3 for ensemble accuracy estimation | Human-machine ensembles · ImageNet-16H and CIFAR10H | absolute estimation error: 0.032 vs 0.624; 0.172 vs 0.259 | Helped |
| Increase ensemble size | Homogeneous human or machine ensembles · ImageNet-16H and CIFAR10H | accuracy | Helped |
| Increase ensemble size beyond the knee point | Homogeneous human or machine ensembles · ImageNet-16H and CIFAR10H | accuracy growth rate | Hurt |
| Add more machines while keeping the human count constant | Mixed human-machine ensembles · ImageNet-16H and CIFAR10H | accuracy | Mixed |
| Follow the optimal human-machine ratio as ensemble size varies | Mixed human-machine ensembles · ImageNet-16H and CIFAR10H | accuracy | Helped |
| Use ACC1 or ACC2 to predict ensemble accuracy instead of ACC3 | Human-machine ensembles · ImageNet-16H and CIFAR10H | accuracy estimation precision | Helped |
Surprising findings
- Mixed human-machine ensembles do not improve monotonically with added agents; adding more of one type can hurt, creating an optimal ratio that shifts with ensemble size.
- On ImageNet-16H, optimal-path accuracy drops at M* in {7,8,9}, attributed to limited numbers of humans and machines.
Limitations
- Relies on Gaussian and conditional-independence assumptions, and assumes consistent bias and variance across agents.
- Experiments restricted to two image classification datasets with a small pool of agents (e.g., 6 humans and 5 machines at M=10 on ImageNet-16H).
- Estimated knee and optimal trajectory can deviate from actual values on some datasets (e.g., CIFAR10H).
Experimental setup
Datasets: CIFAR10H CIFAR10 ImageNet-16H
Benchmarks: CIFAR10H ImageNet-16H
Metrics: ensemble accuracy mean absolute estimation error knee point location
Baselines: Steyvers et al. 2022 accuracy estimate (ACC3)
Builds on: Error-ambiguity decomposition Steyvers et al. 2022 Bayesian human-AI combination Satopaa knee detection method
Closest papers by meaning
- A Mathematical Framework for AI-Human Integration in Work (ICML 2025 · similarity 0.87)
- Position: Towards Bidirectional Human-AI Alignment (NeurIPS 2025 · similarity 0.86)
- Position: AI Scaling: From Up to Down and Out (ICML 2025 · similarity 0.85)
- Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems (NeurIPS 2024 · similarity 0.85)
- Position: Challenges and Future Directions of Data-Centric AI Alignment (ICML 2025 · similarity 0.85)
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.