Research map / Architectures and efficiency / RNNs and normalization
A Model for Estimating the Economic Costs of Computer Vision Systems That Use Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a regression-based economic cost model that estimates the data, compute, and dollar cost needed for transfer-learning computer vision systems to reach a target accuracy.
Problem
Deep learning computer vision projects have unpredictable costs in compute and data, making investment decisions under uncertainty difficult for firms.
Approach
Combines deep learning scaling laws and transfer learning regularities (re-analyzed from Mensink et al. 2021) in a fitted GLM on error vs. datapoints, relative task complexity, pretraining and task type, then converts the solved data requirement into training and data acquisition costs.
Key results
- The fitted GLM (OLS on log10 error) achieves R² of 0.840 with all variables significant at p<0.05.
- On six held-out IBM classification deployments, predicted error matched observed error approximately in all but one case (e.g., 45% observed vs 1% predicted for one Navtech case).
- Example tool output for a 96.5% target accuracy suggests EfficientNet-B0 as lowest-cost model at about $103,034 total cost ($9,611 training plus $93,423 data).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of target-task datapoints | Economic model GLM for transfer learning · Transfer-learning data | log10(error): -0.070 | Helped |
| Use pretraining as datapoints increase | Economic model GLM for transfer learning · Transfer-learning data | log10(error): -0.053 | Helped |
| Increase target-task relative complexity | Economic model GLM · Transfer-learning data | log10(error): 0.609 | Hurt |
| Use semantic segmentation rather than classification | Economic model GLM · Transfer-learning data | log10(error): 0.622 | Hurt |
| Remove dataset distance from the regression model (ablation) | Economic model GLM · Transfer-learning data | statistical significance | No clear effect |
| Use the economic model to predict classification error | Economic model · Boston Sci., 50100 datapoints | error: 3% -> 4% | Mixed |
| Use the economic model to predict classification error | Economic model · Boston Sci., 61000 datapoints | error: 1% -> 4% | Hurt |
| Use the economic model to predict classification error | Economic model · IBM CAO, 61000 datapoints | error: 1% -> 5% | Hurt |
| Use the economic model to predict classification error | Economic model · Navtech, 3000 datapoints, RC 0.200 | error: 6% -> 8% | Hurt |
| Use the economic model to predict classification error | Economic model · Navtech, 3000 datapoints, RC 0.348 | error: 12% -> 9% | Helped |
| Use the economic model to predict classification error | Economic model · Navtech, 3000 datapoints, RC 0.403 | error: 45% -> 1% | Hurt |
Surprising findings
- Dataset distance (Wasserstein distance of Inception-v3 embeddings) failed the significance test and was dropped from the model.
- One held-out case had observed error of 45% against a predicted 1%, a major miss.
Limitations
- Only 14 IBM client datapoints, with bootstrapping from semantic segmentation literature data while the client data are classification, so task mismatch.
- Model applies only to computer vision classification and semantic segmentation; does not capture training from scratch, human/fixed costs, or training time.
- Predictions may be unreliable at extreme target performance values due to extrapolation.
Experimental setup
Datasets: Mensink et al. 2021 transfer learning semantic segmentation data IBM client case studies (14 deployments) ImageNet-1000
Benchmarks: Held-out IBM client case studies (6 deployments)
Metrics: R2 observed vs predicted error estimated total cost
Builds on: Mensink et al. 2021 transfer learning study power-law scaling laws (Hestness et al. 2017) Sevilla et al. 2022 training compute estimation
Closest papers by meaning
- Rethinking the Inception Architecture for Computer Vision (CVPR 2016 · similarity 0.83)
- Learning Time/Memory-Efficient Deep Architectures With Budgeted Super Networks (CVPR 2018 · similarity 0.82)
- DeepOBS: A Deep Learning Optimizer Benchmark Suite (ICLR 2019 · similarity 0.82)
- Reinforcement Learning as a Parsimonious Alternative to Prediction Cascades: A Case Study on Image Segmentation (AAAI 2024 · similarity 0.82)
- Cost-of-Pass: An Economic Framework for Evaluating Language Models (ICLR 2026 · similarity 0.81)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.