Research map / Time series / Causal and generative time series
Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that cherry-picking a few datasets from a 13-dataset benchmark can make most forecasting methods appear best-in-class, and that classical methods are more robust to dataset selection than deep learning models.
Problem
Forecasting papers typically evaluate on a small, possibly unrepresentative set of datasets, so the reported superiority of methods can be biased by dataset selection (cherry-picking).
Approach
A framework that ranks each forecasting model over all subsets of a benchmark collection of a given size, measuring how often a model can be reported as top-k when only a cherry-picked subset is used, and how ranks change as datasets are added.
Key results
- Selecting just four datasets, 46% of the 13 methods could be reported as best in class and 77% within the top three.
- With 3 cherry-picked datasets, more than 54% of models could be reported as top-1 and 92% as top-3.
- Classical methods such as ETS show narrower rank IQR than deep learning models like NHITS, TCN and Informer, whose ranks range widely (e.g., 1 to 13).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select 4 datasets from the benchmark collection | Forecasting model evaluations · 4 cherry-picked datasets | models reported as best: 46\% | Mixed |
| Select 4 datasets from the benchmark collection | Forecasting model evaluations · 4 cherry-picked datasets | models reported in top 3: 77\% | Mixed |
| Select 3 datasets from the benchmark collection | Forecasting model evaluations · 3 cherry-picked datasets | models reported as top 1: more than 54\% | Mixed |
| Select 3 datasets from the benchmark collection | Forecasting model evaluations · 3 cherry-picked datasets | models reported in top 3: 92\% | Mixed |
| Select 6 datasets from the benchmark collection | Forecasting model evaluations · 6 cherry-picked datasets | models reported in top 3: 77\% | Mixed |
| Increase the number of tested datasets from 3 to 6 | Forecasting model evaluations · Dataset subset evaluation | risk of incorrectly identifying an algorithm as best: approximately 40\% | Helped |
| Use cherry-picked datasets for evaluation | Informer and TCN · Selected dataset subsets | model rank | Mixed |
| Evaluate across the full dataset collection rather than a favorable subset | ETS · All benchmark datasets | rank robustness | Helped |
| Evaluate across different datasets | TCN and Informer · All benchmark datasets | rank variability: 1 to 13 | Hurt |
Surprising findings
- Even with 6 datasets, 77% of models could still be reported in the top 3, indicating persistent bias as dataset count grows.
- Simple classical methods such as ETS were more robust than recent deep learning models despite the latter's reported superiority.
Limitations
- Only 13 methods and 9 dataset sources (13 dataset-level series in the summary table) were analyzed, limiting generality.
- Ranking uses SMAPE alone in the framework description, and the paper does not detail training/tuning protocols for baselines.
Experimental setup
Datasets: Labour M3 M4 M5 Tourism Traffic Wiki2 ETTh1 ETTh2
Benchmarks: Labour M3 M4 M5 Tourism Traffic Wiki2 ETTh1 ETTh2
Metrics: SMAPE MASE rank
Baselines: SNaive RWD ETS ARIMA Theta SES Croston RNN TCN DeepAR NHITS TiDE
Builds on: N-BEATS NHITS TiDE DeepAR Informer
Closest papers by meaning
- fev-bench: A Realistic Benchmark for Time Series Forecasting (NeurIPS 2026 · similarity 0.87)
- ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons (NeurIPS 2024 · similarity 0.87)
- Improving Time Series Forecasting via Instance-aware Post-hoc Revision (NeurIPS 2025 · similarity 0.87)
- ChronoEpilogi: Scalable Time Series Selection with Multiple Solutions (NeurIPS 2024 · similarity 0.86)
- SIN: Selective and Interpretable Normalization for Long-Term Time Series Forecasting (ICML 2024 · similarity 0.86)
- Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.