Research map / Label-efficient and robust learning / OOD and anomaly detection

We Need to Rethink Benchmarking in Anomaly Detection

NeurIPS 2026 · Label-efficient and robust learning › OOD and anomaly detection · approach: Anomaly detection with contaminated data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position paper arguing that anomaly detection benchmarking should be reorganized around scenarios defined by shared structural properties, with scenario-specific preprocessing, model selection, ensembles, metrics, and protocols, illustrated by a trivial quantile-based detector that rivals SOTA on ADBench.

Problem

Anomaly detection progress appears to stagnate because aggregated benchmarks reward trivial baselines and fail to reflect the diversity of applications, making algorithm selection unreliable for practitioners.

Approach

Proposes defining anomaly detection scenarios as groups of applications sharing structural properties (anomaly type, modality, dimensionality, supervision, processing time, evaluation priority, operational constraints), and conducting component-wise and metric-wise evaluation within each scenario rather than aggregating across all datasets.

Key results

• A trivial Quantiles detector (fraction of features outside 5%-95% training quantiles) ranks second on ADBench by average ROC-AUC rank, statistically indistinguishable from DeepSVDD, GoAD, and OCSVM; only DTE ranks better.
• The Quantiles detector fails on simple cases such as anomalies within an annulus of normal points.
• The paper proposes a table of eight structural properties (e.g., anomaly type, anomaly ratio, data modality, processing time) to define scenarios.

Surprising findings

• A trivial per-feature quantile detector performs competitively with state-of-the-art deep anomaly detectors on ADBench.
• The only two neural benchmark methods in an earlier unsupervised benchmark (DeepSVDD, DAGMM) ranked last and second-last for global anomalies.

Limitations

• No new benchmark, dataset, or algorithm is released; the scenario taxonomy is proposed but not fully instantiated or validated.
• Quantile experiment is a single illustrative comparison using standard hyperparameters on ADBench.
• Scenario definitions and minimum-coverage guardrails are not operationalized with concrete thresholds.

Experimental setup

Datasets: ADBench

Benchmarks: ADBench

Metrics: ROC-AUC average rank critical difference diagram

Baselines: kNN Isolation Forest DeepSVDD DAGMM GoAD DTE OCSVM LOF Extended Isolation Forest

Builds on: ADBench TSB-AD BMAD

Closest papers by meaning

• The Elephant in the Room: Towards A Reliable Time-Series Anomaly Detection Benchmark (NeurIPS 2024 · similarity 0.88)
• Position: Quo Vadis, Unsupervised Time Series Anomaly Detection? (ICML 2024 · similarity 0.87)
• InCLAD: A Continual Learning Benchmark for Industrial Visual Anomaly Detection (NeurIPS 2026 · similarity 0.86)
• ResAD: A Simple Framework for Class Generalizable Anomaly Detection (NeurIPS 2024 · similarity 0.85)
• Contamination-Resilient Anomaly Detection via Adversarial Learning on Partially-Observed Normal and Anomalous Data (ICML 2024 · similarity 0.85)
• On Diffusion Modeling for Anomaly Detection (ICLR 2024 · similarity 0.85)