Research map / Learning theory and algorithms / Prediction and decision losses
When to retrain a machine learning model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates the retraining decision problem with retraining cost vs. performance cost and proposes UPF, which forecasts future model performance with Beta-distributed uncertainty and decides retraining via quantile comparisons of forecast costs.
Problem
Deciding when to retrain a machine learning model under distribution shift, balancing retraining cost against the cost of degraded performance, given very limited offline data.
Approach
Model future performance entries pe_{i,j} as Beta-distributed random variables predicted by a compact regression (ElasticNetCV) from features (train index, eval index, gap, shift statistics), then retrain when the 95% quantile of the predicted retrain-branch cost is lower than the keep-branch cost, computed recursively over the remaining horizon.
Key results
- UPF achieves the lowest AUC of the combined cost on 5 of 7 datasets, e.g., iWild 3.0498 vs. next best CARA per. 3.7527 and electricity 2.5782 vs. CARA cumul. 2.7147 (significant on most).
- On electricity at alpha=0.1, UPF reaches 0.75 average accuracy with cost 2.24 vs. oracle 0.83 accuracy and 1.93 cost.
- Ablation: 95% quantile UPF beats deterministic PF (50% quantile) on most datasets, e.g., circles 0.0670 vs 0.0697.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UPF instead of CARA cumul. | Retraining decision methods · electricity | AUC of combined cost: 2.7147 -> 2.5782 | Helped |
| Use UPF instead of CARA | Retraining decision methods · Gauss | AUC of combined cost: 0.3841 -> 0.3829 | Helped |
| Use UPF instead of CARA | Retraining decision methods · circles | AUC of combined cost: 0.0726 -> 0.0668 | Helped |
| Use UPF instead of CARA | Retraining decision methods · airplanes | AUC of combined cost: 2.2753 -> 2.2865 | Hurt |
| Use UPF instead of CARA | Retraining decision methods · yelpCHI | AUC of combined cost: 0.1294 -> 0.1293 | Helped |
| Use UPF instead of CARA | Retraining decision methods · epicgames | AUC of combined cost: 0.3202 -> 0.3189 | Helped |
| Use UPF instead of CARA per. | Retraining decision methods · iWild | AUC of combined cost: 3.7527 -> 3.0498 | Helped |
| Add uncertainty-aware 95% quantile decisions instead of deterministic 50% quantile decisions (ablation) | UPF · electricity | AUC of combined cost: 2.5884 ± 0.13 -> 2.6056 ± 0.14 | Hurt |
| Add uncertainty-aware 95% quantile decisions instead of deterministic 50% quantile decisions (ablation) | UPF · Gauss | AUC of combined cost: 0.3673 ± 0.03 -> 0.3643 ± 0.03 | Helped |
| Add uncertainty-aware 95% quantile decisions instead of deterministic 50% quantile decisions (ablation) | UPF · circles | AUC of combined cost: 0.0697 ± 0.01 -> 0.0670 ± 0.01 | Helped |
| Add uncertainty-aware 95% quantile decisions instead of deterministic 50% quantile decisions (ablation) | UPF · airplanes | AUC of combined cost: 2.3688 ± 0.35 -> 2.2688 ± 0.26 | Helped |
| Add uncertainty-aware 95% quantile decisions instead of deterministic 50% quantile decisions (ablation) | UPF · yelpCHI | AUC of combined cost: 0.1180 ± 0.00 -> 0.1175 ± 0.00 | Helped |
Surprising findings
- Distribution shift detection baselines do well only when retraining is cheap; retraining after every detected shift is too frequent when costs are high.
- The oracle retrains 5.6 times on average at low alpha on electricity, while CARA and CARA cumul. retrain 0 or 0.33 times at high cost, illustrating that the CARA baselines are not robust to assumption violations.
Limitations
- Assumes the cost parameter alpha is known or approximated; sensitivity analysis only partially addresses misspecification.
- Each dataset is handled separately; transferring schedulers across datasets is left to future work.
- Performance bound L is hard to estimate in general and empirically approximated from the observed maximum.
Experimental setup
Datasets: electricity airplanes yelpCHI epicgames Gauss circles iWildCam
Benchmarks: electricity Gauss circles airplanes yelpCHI epicgames iWildCam
Metrics: AUC of combined cost C_alpha over alpha range average accuracy number of retrains Wilcoxon test significance
Baselines: ADWIN-5% ADWIN-50% KSWIN-5% KSWIN-50% FHDDM-5% FHDDM-50% CARA CARA cumul. CARA per. oracle
Builds on: CARA ADWIN KSWIN FHDDM offline reinforcement learning
Closest papers by meaning
- Distilled Datamodel with Reverse Gradient Matching (CVPR 2024 · similarity 0.87)
- RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability (AAAI 2025 · similarity 0.87)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.86)
- Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing (NeurIPS 2025 · similarity 0.84)
- RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget (NeurIPS 2025 · similarity 0.84)
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.