Research map / Learning theory and algorithms / Prediction and decision losses
Outlier Ranking for Large-Scale Public Health Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes multi-stream outlier ranking and the Outshines algorithm, which ranks univariate outlier scores across large numbers of public health streams using hierarchical sibling block maxima and extreme value analysis, deployed at Delphi.
Problem
Univariate outlier detection on large-scale public health data streams returns thousands of maximally tied outliers, most of them false positives, so experts cannot review the most important ones.
Approach
Treats outlier ranking as a post-processing step over any univariate detector's test statistics, building an empirical reference distribution per indicator and day by block maxima over sibling streams in a 28-day regime within the geospatial hierarchy, then scoring each point by its quantile.
Key results
- Outshines x EWMA achieved mean AUC of 0.95 versus 0.78 for the next best combination (Outshines x FlaSH) in expert-match evaluation.
- Outshines reduced maximally tied outliers versus Sibling ranking by over 10x (typically 6-18 ties vs. 200-3k).
- Deployed experts identified outliers worth investigating at 0.43 events per minute, a 9.13x improvement over the 0.033 manual baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Outshines ranking with EWMA instead of the next-best combination | Expert evaluation combinations · Expert evaluation snapshot | ROC AUC: 0.78 -> 0.95 | Helped |
| Use Outshines ranking with EWMA instead of other ranking-method combinations | Expert evaluation combinations · Expert evaluation snapshot | ranking correlation: 0.13 -> 0.73 | Helped |
| Replace Sibling ranking with Outshines ranking (ablation) | EWMA outlier ranking · Delphi daily deployment | maximally tied outliers: 14k ± 1.7k -> 21 ± 5.5 | Helped |
| Replace random maximally tied Sibling-ranked points with top-ranked Outshines points | Expert outlier review · One-week direct expert comparison | outliers found: 4.02x | Helped |
| Replace manual review without an outlier detection method with Outshines × EWMA | Delphi expert review · Delphi daily deployment | rate of identifying outliers worth investigating: 9.13 ± 2.26x | Helped |
| Group sibling streams when forming the ranking reference distribution (ablation) | Multi-stream outlier ranking · Expert evaluation snapshot | expert-ranking performance | Helped |
| Use extreme values across hierarchical regions rather than sibling-stream ranking (ablation) | Multi-stream outlier ranking · Expert evaluation snapshot | expert-ranking performance | Helped |
Surprising findings
- Deep learning univariate methods (Telemanom, DeepLog) performed worse than simple EWMA when paired with ranking methods on this task.
Limitations
- Evaluation relies on expert rankings from a single snapshot of 6383 streams and a small number of reviewed streams per respondent.
- Tailored EWMA univariate method is specific to the expert priorities in summer 2023 and may not match future needs.
- Traditional metrics and deployed gains are measured within one public health organization (Delphi).
Experimental setup
Datasets: Delphi public health data streams (COVID-19 indicators) expert-labeled outlier rankings from 17 experts
Benchmarks: Delphi snapshot dataset with expert labels (July 27, 2023)
Metrics: Accuracy F1 ROC AUC Swap correlation Hamming distance number of ties events identified per minute
Baselines: Threshold ranking Optimized Threshold ranking Sibling ranking FlaSH Telemanom DeepLog Isolation Forest Autoregressive model
Builds on: block maxima Sibling ranking EWMA FlaSH TODS
Closest papers by meaning
- Data Monitoring for Large Scale Public Health Data (AAAI 2025 · similarity 0.84)
- UniOD: A Universal Model for Outlier Detection across Diverse Domains (ICLR 2026 · similarity 0.83)
- Tight Rates in Supervised Outlier Transfer Learning (ICLR 2024 · similarity 0.82)
- ALTBI: Constructing Improved Outlier Detection Models via Optimization of Inlier-Memorization Effect (AAAI 2025 · similarity 0.81)
- DOS: Diverse Outlier Sampling for Out-of-Distribution Detection (ICLR 2024 · similarity 0.80)
- Capturing the denoising effect of PCA via compression ratio (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.