Research map / Label-efficient and robust learning / OOD and anomaly detection
Early Concept Drift Detection via Prediction Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Error rate-based concept drift detectors fail to detect early-stage drift when the data distribution changes but the classifier's error rate stays constant.
Approach
Use the prediction uncertainty index u_i = 1 - f_{y_i}(x_i) instead of the error rate as the drift signal; theoretically show error-rate changes imply PU-index changes, and detect drift by Pearson chi-square tests on histograms built with an Adaptive PU-index Bucketing (Ei-kMeans based) partitioning over sliding-window cut points.
Key results
- PUDD ranks first in 17 of 24 incremental-training cases across datasets and classifiers, and in the top 3 in 20 of 24.
- On the SOTA comparison, PUDD attains top rank in 7 of 8 cases; PUDD-5 reaches 98.49% on SEA0, about 2.8% above the best SOTA method.
- The authors report PUDD outperforms classic detectors and Ei-kMeans partitioning on average accuracy across 9 datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · elec2 | accuracy: 76.42 -> 77.28 | Helped |
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · mixed | accuracy: 87.63 -> 89.51 | Helped |
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · ps (powersupply) | accuracy: 72.48 -> 72.79 | Helped |
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · sea0 | accuracy: 97.66 -> 98.49 | Helped |
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · sine | accuracy: 92.18 -> 94.76 | Helped |
| Use PUDD instead of the strongest listed SOTA baseline | PUDD · airline | accuracy: 70.00 -> 64.62 | Hurt |
| Replace Ei-kMeans with Adaptive PU-index Bucketing (ablation) | PUDD · 9 datasets, 3 classifiers | average accuracy | Helped |
| Use the stricter alarm threshold 10^-5 rather than 10^-1 (ablation) | PUDD · classic-detector experiments | top-1 cases: 5, 5, and 8 cases respectively | Helped |
Surprising findings
- On the airline dataset, SOTA ensemble methods NS and ADLTER outperform PUDD, which the authors attribute to tabular, attribute-rich data favoring tree-based ensembles.
Limitations
- Drift alarm threshold must be set manually; the authors note fixed thresholds may not stay optimal over time.
- Retraining on recent data only may underperform on attribute-rich tabular data such as airline.
- Theoretical claims rely on specific histogram constructions and counterexamples rather than a general detection guarantee.
Experimental setup
Datasets: Airline Elec2 PowerSupply Sine Mixed SEA CIFAR-10-CD
Benchmarks: Airline Elec2 PowerSupply Sine Mixed SEA CIFAR-10-CD
Metrics: accuracy rank
Baselines: ADWIN DDM EDDM HDDM-A HDDM-W KSWIN PH MCDD AMF IWE NS ADLTER
Builds on: ADWIN Ei-kMeans DDM Hoeffding tree (VFDT)
Closest papers by meaning
- Identifying Predictions That Influence the Future: Detecting Performative Concept Drift in Data Streams (AAAI 2025 · similarity 0.86)
- Quilt: Robust Data Segment Selection against Concept Drifts (AAAI 2024 · similarity 0.84)
- Online Boosting Adaptive Learning under Concept Drift for Multistream Classification (AAAI 2024 · similarity 0.83)
- RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget (NeurIPS 2025 · similarity 0.83)
- Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift (NeurIPS 2024 · similarity 0.82)
- A Data-Driven Measure of Relative Uncertainty for Misclassification Detection (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.