Research map / Time series / Foundation models for time series
Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a loss-guided structured pruning of time series foundation models (TSFMs) before fine-tuning, so the pruned, sparser model fine-tunes better and can beat specialized PatchTST on full-shot forecasting.
Problem
Fine-tuned time series foundation models often fail to consistently outperform small specialized forecasters (e.g., PatchTST) when full-shot downstream training data is available, and it is unclear how to adapt TSFMs effectively for a target forecasting task.
Approach
Empirical analysis shows pretrained TSFMs have inherent sparsity (inactive FFN channels, insignificant attention heads). The method prunes input/output channels of all linear layers using a Taylor/Fisher-based loss-change importance score computed progressively on downstream batches, then fine-tunes the pruned model, framing pruning as architectural specialization that regularizes fine-tuning rath
Key results
- Prune-then-finetune beats direct fine-tuning in 83% of forecasting tasks, with average error reductions of 1.0% to 4.4% per TSFM and a maximum reduction of 22.8% (Time-MoE_base on ETTm2).
- Win rate against PatchTST rises from 46% to 68% for Time-MoE_base and from 80% to 98.3% for Chronos-bolt-base.
- Pruned Moirai_large achieves up to 7.4x inference speedup on CPU for 720-step forecasting on Weather.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply structured pruning before fine-tuning instead of fine-tuning the original TSFM | Time-series foundation models · Long-term forecasting tasks across the evaluated benchmarks | forecasting error: 83% of forecasting tasks; average error reduction of 4.4%, 4.0%, 2.5%, 3.2%, 1.3%, 3.3%, 1.0% for Time-MoE-base, Time-Mo | Helped |
| Apply structured pruning before fine-tuning | Chronos-bolt-base · ETTh1 (avg. horizons) | MSE: 0.422 -> 0.421 | Helped |
| Transfer a pruning structure selected on ETTh1 to ETTh2 | Chronos-bolt-base · ETTh1 → ETTh2 (horizon 720) | MSE: 0.420 -> 0.389 | Helped |
| Apply pruning before fine-tuning | Chronos · M4 Daily (H=14) | MSE: 511617 -> 536501 | Hurt |
| Use LoRA rather than full fine-tuning after pruning (ablation) | Chronos-bolt-base · ETTh1 (horizon 720) | MSE: 0.451 vs 0.451 | No clear effect |
| Use LoRA instead of full fine-tuning (ablation) | TTM-A · ETTh1 (avg. horizons) | MSE: 0.402 -> 0.403 | Hurt |
Surprising findings
- Up to 90% of parameters can be removed while performance still improves in some settings.
- Parameter-efficient LoRA does not outperform full fine-tuning in most cases.
Limitations
- Full-shot experiments only; no error bars due to compute cost of repeated runs.
- Pruning gains vary across models and datasets, and some settings show performance drops of up to 1%.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity M4
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity M4
Metrics: MSE MAE win rate inference time
Baselines: PatchTST full fine-tuning (FT) zero-shot TSFM LoRA
Builds on: LLM-Pruner Chronos PatchTST
Closest papers by meaning
- Multi-Scale Finetuning for Encoder-based Time Series Foundation Models (NeurIPS 2025 · similarity 0.91)
- Towards Neural Scaling Laws for Time Series Foundation Models (ICLR 2025 · similarity 0.89)
- SEMPO: Lightweight Foundation Models for Time Series Forecasting (NeurIPS 2025 · similarity 0.89)
- TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster (NeurIPS 2025 · similarity 0.89)
- In-Context Fine-Tuning for Time-Series Foundation Models (ICML 2025 · similarity 0.88)
- Exploring Representations and Interventions in Time Series Foundation Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.