Research map / Time series / Causal and generative time series

fev-bench: A Realistic Benchmark for Time Series Forecasting

NeurIPS 2026 · Time series › Causal and generative time series · approach: Robust forecasting and evaluation

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces fev-bench, a 100-task forecasting benchmark across seven domains (46 with covariates) with bootstrap-based win-rate and skill-score aggregation, plus the fev library for reproducible evaluation; finds covariate-aware models gain substantially and most TSFMs ignore covariates.

Problem

Existing time series forecasting benchmarks have limited domain coverage, rarely include covariates, use aggregation without statistical rigor, and lack flexible, reproducible evaluation infrastructure.

Approach

A covariate-inclusive, multi-domain benchmark of 100 rolling-origin tasks evaluated with MASE and SQL, aggregated via average win rate and geometric-mean skill score with paired bootstrap confidence intervals, supported by a lightweight evaluation library that excludes model implementations and provides adapters to popular forecasting libraries.

Key results

• Chronos-2 achieves the highest average win rate (91.4%) and skill score (47.3%) under SQL across the 100 tasks, with bootstrap CIs showing a significant gap over other models.
• On 42 dynamic-covariate tasks, Chronos-2 skill improves from 40.9% to 47.0% and TabPFN-TS from 34.1% to 42.5% when covariates are used.
• TiRex and TimesFM-2.5 have overlapping confidence intervals (win rates 83.3% vs 79.6% under SQL), so no clear second-best model.

Surprising findings

• Multivariate modeling gives only modest gains for Chronos-2 (55.0% vs 54.4%) and Toto-1.0 (54.7% vs 54.1%) on 35 multivariate tasks.
• Most pretrained TSFMs ignore covariates, despite covariates substantially improving the models that use them.

Limitations

• Data leakage is handled via self-reported training overlap rather than leakage-free verification.
• Broad coverage means results may not reflect performance on specialized domain tasks.
• Some models (Stat. Ensemble, AutoARIMA, AutoETS) fail on tasks exceeding a 6-hour time limit.

Experimental setup

Datasets: GIFT-Eval Monash BOOM / BOOMLET Kaggle competition datasets (Favorita, M5, Rossmann, Walmart, Rohlik) FRED-MD FRED-QD ENTSO-e load EPF

Benchmarks: fev-bench

Metrics: MASE SQL average win rate skill score pairwise win rate pairwise skill score WQL WAPE

Baselines: Seasonal Naive AutoETS AutoARIMA AutoTheta SCUM ensemble CatBoost LightGBM PatchTST TFT DeepAR

Builds on: Monash repository GIFT-Eval BOOM Hugging Face datasets SCUM ensemble

Closest papers by meaning

• Improving Time Series Forecasting via Instance-aware Post-hoc Revision (NeurIPS 2025 · similarity 0.87)
• Cherry-Picking in Time Series Forecasting: How to Select Datasets to Make Your Model Shine (AAAI 2025 · similarity 0.87)
• Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series (NeurIPS 2025 · similarity 0.86)
• ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons (NeurIPS 2024 · similarity 0.86)
• TimePFN: Effective Multivariate Time Series Forecasting with Synthetic Data (AAAI 2025 · similarity 0.86)
• Physiome-ODE: A Benchmark for Irregularly Sampled Multivariate Time-Series Forecasting Based on Biological ODEs (ICLR 2025 · similarity 0.86)