Research map / Time series / Foundation models for time series
Exploring Representations and Interventions in Time Series Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes layer-wise representation similarity (CKA) in time series foundation models, uses it for block-wise layer pruning, and steers TSFM outputs (trend/periodicity) by adding concept vectors to hidden activations.
Problem
Internal representations of time series foundation models (TSFMs) are poorly understood; it is unclear how their layers are organized, where human-interpretable concepts (trends, periodicity) are encoded, and whether they can be steered at inference time.
Approach
Applies CKA-based representation similarity to TSFMs to reveal block-like layer redundancy and prunes whole redundant blocks (keeping boundary layers); localizes linearly separable concepts with Fisher LDR over layers and tokens, and steers predictions by adding median-difference concept vectors to hidden states without retraining.
Key results
- Pruning only Block 3 of MOMENT-Large reduced estimated model size by 11% and inference time by 5% while lowering zero-shot imputation MAE on 5 of 9 datasets.
- Fine-tuned pruned MOMENT-Large with all block redundancies removed matched the unpruned baseline MSE on ETT datasets and ILI (e.g., ILI 2.981 vs 3.260), reducing memory by over 50%.
- Concept steering with a steering matrix shifted constant inputs toward periodic or trending outputs; on ECG5000 all 30 steered normal samples switched class.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prune all redundant blocks (ablation) | MOMENT-Large · Long-horizon forecasting | MSE: 0.371 -> 0.385; 0.132 -> 0.132; 0.204 -> 0.290; 0.080 -> 0.171; 3.260 -> 2.981; 0.153 -> 0.152 | Mixed |
| Prune only Block 3 (ablation) | MOMENT-Large · Zero-shot imputation, ETTh1 | MSE: 0.371 -> 0.424 | Hurt |
| Prune only Block 3 (ablation) | MOMENT-Large · Zero-shot imputation, ETTh2 | MSE: 0.132 -> 0.133 | Hurt |
| Prune only Block 3 (ablation) | MOMENT-Large · Zero-shot imputation, ETTm1 | MSE: 0.204 -> 0.221 | Hurt |
| Prune only Block 3 (ablation) | MOMENT-Large · Zero-shot imputation, ETTm2 | MSE: 0.080 -> 0.076 | Helped |
| Use mean rather than median activation differences for steering (ablation) | TSFM concept steering · Synthetic concept steering | steered output | No clear effect |
| Apply steering to a single token rather than all tokens (ablation) | TSFM concept steering · Synthetic concept steering | concept steering success | Hurt |
Surprising findings
- Scaling within a model family does not produce predictable changes in representation block structure.
- Concept steering effect does not depend notably on using mean versus median for the steering matrix.
Limitations
- Analysis limited to transformer-based TSFMs; state space models and MLP architectures not studied.
- Steering evaluated mainly on synthetic data and tasks limited to forecasting and imputation.
- Redundant blocks identified by visual inspection of heatmaps rather than a validated algorithm.
Experimental setup
Datasets: synthetic constant and sinusoidal time series ETT Informer long-horizon datasets UCR Time-Series Archives Monash Forecasting Archive Time Series Pile ECG5000
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 ILI Weather Electricity Exchange rate Traffic Monash forecasting archive
Metrics: MSE MAE MASE WQL inference time model size
Baselines: unpruned MOMENT-Large unpruned Chronos-Large Moirai-1.1-R Large
Builds on: Centered Kernel Alignment SVCCA linear probing
Closest papers by meaning
- Towards Neural Scaling Laws for Time Series Foundation Models (ICLR 2025 · similarity 0.90)
- Less is More: Unlocking Specialization of Time Series Foundation Models via Structured Pruning (NeurIPS 2025 · similarity 0.88)
- Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Models (NeurIPS 2026 · similarity 0.88)
- Multi-Scale Finetuning for Encoder-based Time Series Foundation Models (NeurIPS 2025 · similarity 0.88)
- Multi-Patch Prediction: Adapting Language Models for Time Series Representation Learning (ICML 2024 · similarity 0.87)
- Empowering Time Series Analysis with Large-Scale Multimodal Pretraining (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.