Research map / Label-efficient and robust learning / Test-time adaptation
WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes weighted-conformal test martingales (WCTMs) and the WATCH monitoring framework that adapts online to benign covariate shifts while alarming on harmful concept or extreme covariate shifts, with false-alarm control.
Problem
Post-deployment monitoring of AI/ML systems needs to avoid false alarms under benign shifts, quickly detect harmful shifts, and diagnose their cause, but standard conformal test martingales assume exchangeability and cannot adapt.
Approach
Generalizes conformal test martingales to weighted-conformal p-values (weights from density-ratio estimates), proving independence and exact validity under a bag-sufficiency condition; WATCH runs a weighted CTM alongside a covariate-only X-CTM that triggers adaptation and enables root-cause diagnosis.
Key results
- On MEPS benign covariate shift (averaged over 200 seeds), WCTMs avoid unnecessary alarms that standard CTMs raise while maintaining 0.9 coverage.
- Anytime-valid WCTMs and CTMs achieve comparable detection delay, with (W)CTMs over three times faster than PR-ST baseline.
- On CIFAR-10 brightness corruption, WCTM avoids alarms at mild corruption levels and raises an alarm only at the severest level tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WCTM instead of standard CTM for benign covariate-shift monitoring | WATCH on MEPS · MEPS benign covariate shift | unnecessary alarms | Helped |
| Adapt prediction intervals with WCTM under benign covariate shift | WATCH on MEPS · MEPS benign covariate shift | interval width | Helped |
| Use WCTM instead of standard CTM under benign covariate shift | WATCH on MEPS · MEPS benign covariate shift | coverage | No clear effect |
| Use WCTM instead of standard CTM for level-1 brightness corruption with over 50% clean samples | WATCH · CIFAR-10 brightness corruption | unnecessary alarms | Helped |
| Increase brightness corruption to level-3 with clean-sample ratio reduced to 40% | WATCH WCTM · CIFAR-10 brightness corruption | alarm | No clear effect |
| Increase brightness corruption to level-5 with clean-sample ratio reduced to 30% | WATCH WCTM · CIFAR-10 brightness corruption | harmful-shift detection | Helped |
| Use WCTM instead of PR-ST for harmful concept-shift monitoring | WATCH · Harmful concept shifts | average detection delay: over three times faster | Helped |
| Use WCTM instead of standard CTM for harmful-shift detection | WATCH · Harmful concept shifts | average detection delay | No clear effect |
| Use WCTM-based Shiryaev-Roberts instead of PR-CD for harmful-shift detection | WATCH · Harmful concept shifts | average detection delay | Helped |
| Use online PR-CD instead of minibatch PR-CD | PR-CD · Harmful concept shifts | average detection delay | Helped |
| Run PR-CD online rather than use WCTM or SR-WCTM | Sequential harmful-shift monitoring · Harmful concept shifts | wall-clock runtime: O(t^2) versus O(t) | Hurt |
| Add a parallel X-CTM to WCTM for root-cause diagnosis | WATCH · Benign and harmful shifts | root-cause diagnosis | Helped |
Surprising findings
- Earlier versions of the theorem omitted bag sufficiency and were incorrect; a three-observation counterexample shows marginal uniformity of each p-value does not imply independence.
Limitations
- Theory requires bag sufficiency and correct density-ratio estimates for exact validity; approximate weights lack robustness results.
- Experiments are limited to a small set of tabular and image datasets with synthetic shifts.
Experimental setup
Datasets: MEPS MNIST CIFAR-10 CIFAR-10-C
Benchmarks: MEPS CIFAR-10-C MNIST
Metrics: empirical coverage interval width average detection delay false alarm rate average run length
Baselines: standard conformal test martingales PR-ST PR-CD
Builds on: conformal test martingales weighted conformal prediction composite jumper betting Ville's inequality online compression models
Closest papers by meaning
- Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them) (ICML 2024 · similarity 0.86)
- Online Conformal Prediction via Online Optimization (ICML 2025 · similarity 0.84)
- Monitoring Risks in Test-Time Adaptation (NeurIPS 2025 · similarity 0.84)
- Adaptive auditing of AI systems with anytime-valid guarantees (NeurIPS 2026 · similarity 0.83)
- Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach (NeurIPS 2024 · similarity 0.83)
- Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMs (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.