Research map / Label-efficient and robust learning / OOD and anomaly detection
Unsupervised Anomaly Detection in The Presence of Missing Values
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ImAD, an end-to-end unsupervised anomaly detection method that jointly learns imputation and detection on incomplete data by generating pseudo-abnormal samples in a latent space to counter imputation bias.
Problem
Unsupervised anomaly detection on incomplete data with high missing rates is hard because "impute-then-detect" pipelines trained only on normal data produce imputation bias that makes abnormal samples look normal.
Approach
Jointly optimize imputer, projector, and reconstructor in one objective, and generate pseudo-abnormal samples from a hypershell around a normal latent ball so the imputer generalizes to abnormal incomplete data; includes theoretical generalization and detection guarantees.
Key results
- On Adult under MCAR with 20% missingness, ImAD reaches 76.51% AUROC versus the best baseline (e.g., GAIN+NeutraL AD 90.48% is on KDD; on Adult the strongest baselines are lower), with ImAD achieving higher AUROC in almost all manually-masked settings.
- On inherently missing datasets, ImAD attains 82.09% AUROC on Titanic, 66.32% on MovieLens1M, 100% on Bladder, and 96.62% on Seq2-Heart, outperforming all baselines in every case.
- Ablation shows removing pseudo-abnormal samples lowers ImAD AUROC across datasets (e.g., Adult 65.30 vs 76.51 at 20% missing).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ImAD instead of the strongest listed baseline | ImAD · KDD, MCAR, mr=0.2 | AUROC: 96.63(1.05) -> 97.01(0.33) | Helped |
| use ImAD instead of the strongest listed baseline | ImAD · KDD, MCAR, mr=0.5 | AUROC: 93.37(2.15) -> 90.78(0.09) | Hurt |
| use ImAD instead of the strongest listed baseline | ImAD · Adult, MCAR, mr=0.2 and mr=0.5 | AUROC: 64.39(0.15) -> 76.51(2.12); 65.44(2.40) -> 71.19(1.63) | Helped |
| use ImAD instead of the strongest listed baseline | ImAD · Arrhythmia, MCAR, mr=0.2 and mr=0.5 | AUROC: 80.79(1.11) -> 82.24(1.76); 82.64(1.27) -> 81.76(1.19) | Mixed |
| use ImAD instead of the strongest listed baseline | ImAD · Speech, MCAR, mr=0.2 and mr=0.5 | AUROC: 60.37(0.87) -> 61.94(2.77); 29.09(1.14) -> 58.66(1.40) | Helped |
| use ImAD instead of the strongest listed baseline | ImAD · Titanic, MovieLens1M, Bladder, and Seq2-Heart with inherent missing values | AUROC: 79.72(0.28) -> 82.09(0.99); 64.84(1.04) -> 66.32 (1.36); 99.99(0.00) -> 100(0.00); 95.31(0.32) -> 96.62(0.11) | Helped |
| add pseudo-abnormal samples during training (ablation) | ImAD · Adult, MCAR, mr=0.2 | AUROC: 65.30 (4.36) -> 76.51 (2.12) | Helped |
| add pseudo-abnormal samples during training (ablation) | ImAD · KDD, MCAR, mr=0.5 | AUROC: 91.50 (1.59) -> 90.78(1.35) | No clear effect |
| add pseudo-abnormal samples during training (ablation) | ImAD · Titanic, MovieLens1M, and Seq2-Heart with inherent missing values | AUROC: 76.14 (1.66) -> 82.09 (0.99); 60.62(1.35) -> 66.32 (1.36); 95.19 (0.71) -> 96.62 (0.11) | Helped |
| add pseudo-abnormal samples during training (ablation) | ImAD · Bladder, inherent missing values | AUROC: 99.90 (0.21) -> 1.00 (0.00) | No clear effect |
| vary the latent dimension (ablation) | ImAD · Adult, Botnet, KDD, Arrhythmia, Speech, Segerstolpe, and Usoskin | AUROC | Helped |
| use MCAR rather than MAR or MNAR to generate pseudo-abnormal missingness (ablation) | ImAD · Titanic, MovieLens1M, Bladder, and Seq2-Heart | AUROC: 82.09 vs 79.06 and 80.50; 66.32 vs 63.14 and 61.44; 100.00 vs 99.95 and 100.00; 96.62 vs 96.79 and 95.56 | Mixed |
Surprising findings
- More sophisticated imputers (GAIN) often yield worse downstream detection than simple MissForest, attributed to imputation bias.
- Detection performance of impute-then-detect methods sometimes does not decrease, and can even worsen, as missing rate drops from 0.5 to 0.2.
Limitations
- Evaluated only on tabular data; incomplete image data and time series not considered.
- Theoretical guarantees rely on assumptions (spectral and l2,1 norm bounds, Lipschitz activations) and the constant c in detection theorem is not determined for neural networks.
Experimental setup
Datasets: Adult KDD Botnet Arrhythmia Speech Segerstolpe Usoskin Titanic MovieLens1M Bladder Seq2-Heart
Benchmarks: Adult KDD Botnet Arrhythmia Speech Segerstolpe Usoskin Titanic MovieLens1M Bladder Seq2-Heart
Metrics: AUROC AUPRC
Baselines: Isolation Forest + MissForest Isolation Forest + GAIN Deep SVDD + MissForest Deep SVDD + GAIN NeutraL AD + MissForest NeutraL AD + GAIN DPAD + MissForest DPAD + GAIN Mean-Filling Zero-Filling
Builds on: Deep SVDD NeutraL AD Sinkhorn distance MissForest GAIN
Closest papers by meaning
- Distribution Shift in Missing Data Imputation: A Risk-Based Perspective and Importance-Weighted Correction under MAR (NeurIPS 2026 · similarity 0.87)
- Random features models: a way to study the success of naive imputation (ICML 2024 · similarity 0.85)
- Normal-Abnormal Guided Generalist Anomaly Detection (NeurIPS 2025 · similarity 0.85)
- Contamination-Resilient Anomaly Detection via Adversarial Learning on Partially-Observed Normal and Anomalous Data (ICML 2024 · similarity 0.84)
- Unsupervised Anomaly Detection for Tabular Data Using Deep Noise Evaluation (AAAI 2025 · similarity 0.84)
- Towards a Unified Framework of Clustering-based Anomaly Detection (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.