Research map / Large language models / Agents, code and math
Self-Healing Machine Learning: A Framework for Autonomous Adaptation in Real-World Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes self-healing machine learning (SHML), a framework where a healing mechanism diagnoses the reason for model degradation and selects diagnosis-guided corrective actions, instantiated as H-LLM using GPT-4 on simulated tabular data.
Problem
Deployed ML models degrade under distribution shifts, and existing adaptation methods (retraining, reusing old models, drift handlers) are reason-agnostic, choosing pre-defined actions without diagnosing why performance dropped.
Approach
Formalize a four-stage loop (monitoring, diagnosis, adaptation, testing) where the adaptation policy conditions on a diagnosis distribution over reasons for degradation; instantiate it with LLMs that propose diagnoses and candidate actions, which are then selected by backtesting.
Key results
- On simulated diabetes data with k=3 corrupted columns at tau=0.05 corruption, H-LLM reaches about 0.90 accuracy versus about 0.69-0.70 for retraining and ensemble baselines.
- Across five datasets with lagged labels and corrupted features, H-LLM outperforms baselines, e.g. poker accuracy 0.70 vs 0.48-0.49 for the best baselines when k=5.
- Diagnosis KL-divergence to true corruption decreases as outlier factor and corruption coefficient increase, showing diagnosis quality improves when issues are more apparent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use diagnosis-guided adaptation to identify and remove corrupted samples, then retrain on the remaining data, rather than standard reason-agnostic adaptation | H-LLM and deployed logistic regression · Diabetes, corrupted columns and corruption percentages in Table, Viability study I | accuracy | Helped |
| Use diagnosis-guided feature decorruption at test time | H-LLM · Airlines, Poker, Weather, Electricity, Forest Type | accuracy | Helped |
| Lower the drift-detection threshold (increase detection sensitivity) | H-LLM · Diabetes, simulated intervention | post-intervention accuracy | Helped |
| Increase outlier factor and corruption coefficient so data issues are more apparent | H-LLM diagnosis · Diagnosis study, corrupted-variable identification | KL-Divergence | Helped |
| Use a larger backtesting dataset | H-LLM adaptation selection · Adaptation study, varying backtesting-window size | accuracy | Helped |
| Evaluate proposed actions with a testing phase using a backtesting window | H-LLM · Testing study, simulated diabetes task | accuracy | Helped |
Surprising findings
- Reason-agnostic baselines such as new model training and ensembles can perform worse than doing nothing under heavy corruption (e.g. accuracy drops to about 0.50 at k=8 versus 0.45 for no retraining).
Limitations
- Success depends on accurate root-cause identification and effective adaptation policies, which may fail in complex real-world settings.
- H-LLM primarily finds subgroup-level issues; prioritization of adaptation strategies is not trivial.
- Experiments use simulated shifts with logistic regression models and a single LLM (GPT-4).
Experimental setup
Datasets: simulated diabetes prediction data Airlines Poker Weather Electricity Forest Type (CovType)
Benchmarks: simulated diabetes prediction Airlines Poker Weather Electricity Forest Type
Metrics: accuracy KL-divergence of diagnosis average recovery time post-intervention accuracy
Baselines: No retraining Partially updating New model training Ensemble method
Builds on: Drift Detection Method chain-of-thought concept drift adaptation
Closest papers by meaning
- Self-Adapting Language Models (NeurIPS 2025 · similarity 0.83)
- Explorations of Self-Repair in Language Models (ICML 2024 · similarity 0.83)
- ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification (ICML 2025 · similarity 0.83)
- A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops (ICLR 2025 · similarity 0.83)
- Coherence Mechanisms for Provable Self-Improvement (NeurIPS 2026 · similarity 0.82)
- RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.