Research map / Time series / Foundation models for time series
Specialized Foundation Models Struggle to Beat Supervised Baselines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares specialized foundation models in genomics, satellite imaging, and time series against tuned supervised baselines (NAS-tuned CNNs via DASHA and a GPU-tuned linear autoregression, Auto-AR), finding supervised models match or beat most FMs.
Problem
Whether specialized foundation models pretrained on large domain data actually outperform well-tuned traditional supervised learning on the same downstream tasks in genomics, satellite imaging, and time series.
Approach
Builds strong, automated supervised workflows: DASHA (NAS over kernel sizes and dilation rates of wide ResNet/UNet backbones, plus ASHA tuning) and Auto-AR (KPSS/BIC-selected differencing and long lookback AR fit on GPU), and shows these match or beat recent FMs across many tasks without pretraining data.
Key results
- On the NT genomics benchmark, DASHA attains average score 0.761 and average rank 3.69, outperforming all listed foundation models such as Caduceus-PH (0.725) and NT-Multispecies (500M) (0.700).
- On satellite imaging, DASHA reaches 77.85 average score, within about 1 point of the best FM (DOFA-Large 78.80) and above SatMAE-Base (76.99) and CROMA-Base (77.39).
- On seven long-horizon forecasting datasets, Auto-AR achieves 0.551 average RMSE, beating most zero-shot and fine-tuned time series FMs and matching TTM (A) at 0.538 as the top-ranked method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the DASHA supervised workflow instead of the strongest FM | Nucleotide Transformer benchmark · Nucleotide Transformer benchmark | average score: 0.725 -> 0.761 | Helped |
| Use the DASHA supervised workflow instead of DOFA-Large | Satellite imaging tasks · Satellite imaging tasks | average score: 78.80 -> 77.85 | Hurt |
| Use Auto-AR instead of Auto-ARIMA | Long-horizon forecasting · Long-horizon forecasting, seven datasets | average RMSE: 0.896 -> 0.551 | Helped |
| Tune CNN kernel sizes and dilation rates with DASHA | CNN supervised baselines for genomics and satellite imaging · Nucleotide Transformer benchmark and satellite imaging tasks | average score | Helped |
| Allow AR lookback windows longer than five | Auto-AR for time-series forecasting · Long-horizon forecasting | RMSE | Helped |
| Drop the moving-average component and tune AR on GPU | Auto-AR for time-series forecasting · Long-horizon forecasting | RMSE | Helped |
| Increase time-series FM model scale and pretraining data | Time-series foundation models · Long-horizon forecasting | average RMSE | No clear effect |
| Use a supervised baseline with only 20% of the task data | Supervised workflows · Data-scarce regime across genomics, satellite imaging, and time series | performance: 20% | No clear effect |
Surprising findings
- Simply tuning kernel sizes and dilation rates of standard CNNs dominates the genomics benchmark, outperforming all genomics foundation models.
- Untuned or lightly tuned linear autoregression with long lookbacks (up to 512) on GPU outperforms all open-source time series foundation models on average.
Limitations
- Results are lower bounds on supervised performance since model development is approximated via low-fidelity NAS rather than human design.
- Several time series FMs (Time-LLM, Moirai, LLM4TS, Toto) were excluded or reported only partially due to reproducibility or subset issues.
- Evaluation focuses on full-data classification and long-horizon forecasting; FMs may still help in low-data, exploratory, or other tasks.
Experimental setup
Datasets: Nucleotide Transformer benchmark GeoBench BigEarthNet EuroSAT Canadian Cropland fMoW-Sentinel ETT Weather Electricity ILI Traffic
Benchmarks: Nucleotide Transformer benchmark GeoBench long-horizon forecasting datasets (ETT, Weather, Electricity, ILI, Traffic)
Metrics: MCC F1 accuracy top-1 accuracy mAP RMSE average rank mean % improvement median % improvement
Baselines: Enformer DNABERT-2 HyenaDNA Caduceus Nucleotide Transformer SatMAE CROMA DOFA TTM MOMENT GPT4TS TimesFM
Builds on: DASH ASHA Auto-ARIMA wide ResNet UNet
Closest papers by meaning
- Foundation Models Secretly Understand Neural Network Weights: Enhancing Hypernetwork Architectures with Foundation Models (ICLR 2025 · similarity 0.87)
- Scaling Laws for Optimal Data Mixtures (NeurIPS 2025 · similarity 0.86)
- The Best Instruction-Tuning Data are Those That Fit (NeurIPS 2025 · similarity 0.86)
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models (NeurIPS 2025 · similarity 0.86)
- Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition (NeurIPS 2025 · similarity 0.86)
- Towards Neural Scaling Laws for Time Series Foundation Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.