Research map / Large language models / Language, safety and interpretability
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
Connor Toups, Rishi Bommasani, Kathleen Creel, Sarah H Bana, Dan Jurafsky, Percy Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate the ecosystem-wide distribution of outcomes against an independence baseline rather than only individual model accuracy | Commercial ML APIs across 11 HAPI datasets · HAPI | systemic and consistent-success outcomes relative to baseline | Mixed |
| Improve one commercial API | Commercial ML APIs across HAPI datasets · HAPI, 2020–2022 | improvements on systemic failures relative to potential improvements: just 10%; 27% | Hurt |
| Improve Amazon's sentiment API | Amazon sentiment API · Waimai, 2020–2021 | net systemic failures: 2.5%; 78; 78 | No clear effect |
| Use gross improvements instead of net improvements (ablation) | Year-over-year API improvements · HAPI | progress on systemic failures | No clear effect |
| Vary the threshold for including model improvements (ablation) | Year-over-year API improvements · HAPI | under-improvement on systemic failures: 6 different thresholds | No clear effect |
| Include HAM10k predictions (ablation) | Dermatology model ecosystem · DDI | homogeneous outcomes | Hurt |
| Include HAM10k predictions (ablation) | Dermatology model ecosystem · DDI, skin-tone disparity | racial disparity in model homogeneity | Hurt |
| Stratify FER+ examples by annotator disagreement (ablation) | Three commercial FER APIs · FER+ | homogeneous outcomes relative to baseline | No clear effect |
| Stratify DDI examples by dermatologist accuracy (ablation) | Dermatology models · DDI, instances both dermatologists classify correctly | homogeneity relative to baseline | No clear effect |
| Stratify DDI examples by dermatologist accuracy (ablation) | Dermatology models · DDI, instances both dermatologists misclassify | homogeneous outcomes relative to baseline | Improved |
| Replace the instance-level independence baseline with a shared hard/easy-instance baseline (ablation) | Theoretical ecosystem outcome model · HAPI | fit to observed outcome distributions: 0.2-0.3; 1-4 | Improved |
| Examine ecosystems of human dermatologists as well as models | Two board-certified dermatologists · DDI | homogeneity relative to model outcomes | Mixed |
Closest papers by meaning
- Fostering the Ecosystem of AI for Social Impact Requires Expanding and Strengthening Evaluation Standards (NeurIPS 2025 · similarity 0.84)
- Comparing Targeting Strategies for Maximizing Social Welfare with Limited Resources (ICLR 2025 · similarity 0.84)
- Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions (NeurIPS 2024 · similarity 0.83)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.83)
- Position: Human Baselines in Model Evaluations Need Rigor and Transparency (With Recommendations & Reporting Checklist) (ICML 2025 · similarity 0.82)
- A hierarchical decomposition for explaining ML performance discrepancies (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
