Research map / Time series / Anomaly detection
The Elephant in the Room: Towards A Reliable Time-Series Anomaly Detection Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TSB-AD, a curated benchmark of 1070 time series (870 univariate, 200 multivariate) from 40 datasets with 40 anomaly detectors and 10 evaluation measures, recommending VUS-PR as the most reliable measure.
Problem
Time-series anomaly detection evaluation suffers from flawed datasets (mislabeling, run-to-failure bias, triviality), biased measures (e.g., point adjustment favoring random scores, ROC overestimation), and inconsistent benchmarking practices.
Approach
Combine human perception and algorithmic label-quality tests to curate a large heterogeneous dataset, identify VUS-PR as the most lag-robust and unbiased measure, and run a unified, hyperparameter-tuned evaluation of 40 statistical, neural, and foundation-model detectors.
Key results
- Sub-PCA ranks first on TSB-AD-U by VUS-PR, while CNN ranks first on TSB-AD-M, with statistical methods dominating univariate results.
- Fine-tuned MOMENT outperforms zero-shot MOMENT on TSB-AD-U, and foundation models (TimesFM, Chronos) excel at point anomalies, ranking first and third in that analysis.
- Random scores rank 26th under PA-F1 among 32 detectors, showing point adjustment is biased toward noisy inputs; VUS-PR is the most lag-robust measure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VUS-PR rather than the other evaluated measures for anomaly detection evaluation | time-series anomaly detection evaluation · TSB-AD evaluation measure analysis | robustness to lags and evaluation accuracy | Helped |
| use PA-F1 to evaluate a continuous random anomaly score | time-series anomaly detection evaluation · TSB-AD evaluation measure analysis | random-score ranking: 26 | Hurt |
| use statistical methods rather than neural-network methods | TSB-AD-U · TSB-AD-U | VUS-PR ranking | Helped |
| use neural-network methods for multivariate detection | TSB-AD-M · TSB-AD-M | VUS-PR ranking | Helped |
| fine-tune MOMENT rather than use its zero-shot version | MOMENT · TSB-AD-U | VUS-PR ranking | Helped |
| use foundation models for point-based anomaly detection | time-series anomaly detectors · TSB-AD-U point-based anomalies | Friedman/Nemenyi rank | Mixed |
| use statistical methods for sequence-based anomaly detection | time-series anomaly detectors · TSB-AD-U sequence-based anomalies | Friedman/Nemenyi rank | Helped |
| use neural networks for single-anomaly scenarios | time-series anomaly detectors · TSB-AD-U single-anomaly scenarios | Friedman/Nemenyi rank | Hurt |
| use MOMENT for multiple-anomaly scenarios | MOMENT · TSB-AD-U multiple-anomaly scenarios | Friedman/Nemenyi rank | Helped |
| use simpler CNN/LSTM architectures rather than advanced transformer architectures | time-series anomaly detectors · TSB-AD | VUS-PR performance | Helped |
| use an LLM-based approach such as OFA for time-series anomaly detection | time-series anomaly detectors · TSB-AD | VUS-PR performance | Hurt |
Surprising findings
- Simpler architectures such as CNNs and LSTMs generally outperform complex transformer designs, and neural networks do not show the superiority often attributed to them on univariate data.
- LLM-based anomaly detection (OFA) yields unsatisfactory results, indicating a research gap.
Limitations
- Foundation models may be contaminated by anomaly detection datasets in pretraining (data contamination risk).
- Manual inspection relies on a limited number of experienced time-series users; no benchmark is perfect.
Experimental setup
Datasets: UCR NAB YAHOO IOPS MGAB WSD SED TODS NEK Stock Power GHL
Benchmarks: TSB-AD-U TSB-AD-M
Metrics: VUS-PR VUS-ROC AUC-ROC AUC-PR Standard-F1 PA-F1 Event-based-F1 R-based-F1 Affiliation-F1 PATE
Baselines: KMeansAD Sub-PCA KShapeAD OmniAnomaly USAD CNN Donut TranAD
Builds on: TSB-UAD Wu & Keogh flaw analysis VUS measures point adjustment
Closest papers by meaning
- Position: Quo Vadis, Unsupervised Time Series Anomaly Detection? (ICML 2024 · similarity 0.91)
- Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability. (NeurIPS 2026 · similarity 0.90)
- Structured Temporal Causality for Interpretable Multivariate Time Series Anomaly Detection (NeurIPS 2025 · similarity 0.90)
- When Model Meets New Normals: Test-Time Adaptation for Unsupervised Time-Series Anomaly Detection (AAAI 2024 · similarity 0.89)
- Supervision Recovery for Time Series Anomaly Detection via Counterfactual Pairing (NeurIPS 2026 · similarity 0.89)
- Towards a General Time Series Anomaly Detector with Adaptive Bottlenecks and Dual Adversarial Decoders (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.