Research map / Time series / Long-term forecasting
Efficient Time Series Processing for Transformers and State-Space Models through Token Merging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces local token merging for time series transformers and state-space models, a causal, locality-constrained merging scheme that gives linear-to-quadratic complexity and speeds up pretrained forecasters and Chronos without retraining.
Problem
Long token sequences in time series models make transformers and state-space models computationally expensive, and existing token merging (designed for ViTs) is quadratic and non-causal, so it cannot be used for long sequences or decoders.
Approach
A local token merging algorithm that restricts similarity computation to a neighborhood of width k (k=1 gives causal linear-complexity merging), with unmerging for decoders and dynamic, similarity-threshold-based merging; spectral entropy and THD of datasets are shown to predict benefits.
Key results
- Local merging accelerates pretrained transformers up to 3.80x with no MSE change in most settings, finding Pareto-optimal points in 17 of 25 settings.
- On Chronos zero-shot forecasting, local merging improves MSE by up to 9% and speeds up inference by up to 54.76x (reported as up to 5400% acceleration).
- On HyenaDNA, local merging gives 3.62x speed-up with a 4.9% accuracy drop, while global merging drops accuracy by 9.5%; Mamba reaches 4.09x acceleration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply local merging to Chronos at its fastest setting | Chronos · ETTh1 zero-shot forecasting | acceleration: 32.76× | Helped |
| Apply local merging to Chronos at its fastest setting | Chronos · Weather zero-shot forecasting | acceleration: 54.76× | Helped |
| Use less aggressive local merging rather than no merging | HyenaDNA · Dummy Mouse Enhancers Ensembl | accuracy: 78.9% -> 80.6% | Helped |
| Use fastest local merging rather than no merging | HyenaDNA · Dummy Mouse Enhancers Ensembl | accuracy: 78.9% -> 74.0% | Hurt |
| Use local rather than global merging at the fastest setting (ablation) | HyenaDNA · Dummy Mouse Enhancers Ensembl | accuracy: 69.4% -> 74.0% | Helped |
| Use local rather than global merging at the fastest setting (ablation) | HyenaDNA · Dummy Mouse Enhancers Ensembl | acceleration: 2.93× -> 3.62× | Helped |
| Use local merging with the fastest setting | Mamba · Dummy Mouse Enhancers Ensembl | accuracy: 76.0% -> 74.4% | Hurt |
| Apply local merging during training | Autoformer · Traffic | training acceleration: 2.27× | Helped |
| Use dynamic rather than fixed-r merging (ablation) | Chronos · forecasting settings tested | forecast quality | Helped |
| Use dynamic merging with batch size 10 rather than the optimal single-sample case (ablation) | Chronos · batch size 10 | forecast quality | Hurt |
| Use cosine similarity rather than L1 or L2 token similarity (ablation) | Chronos small · ETTh1 | forecast quality | Helped |
| Merge tokens rather than prune them (ablation) | Chronos small · ETTh1 | MSE | Helped |
Surprising findings
- Token merging often improves forecasting MSE while accelerating models, e.g., Chronos and local merging act as an adaptive low-pass filter.
- Gaussian low-pass filtering of inputs yields a similar MSE improvement to local merging on ETTh1.
Limitations
- Merging is restricted to occur only between tokens of two disjoint sets, which limits flexibility.
- Not all hyperparameters were ablated due to the large number of architectures and datasets.
- Some transformer models (e.g., Autoformer on Traffic) show MSE degradation when merging only at inference without training.
Experimental setup
Datasets: ETTh1 ETTm1 Weather Electricity Traffic Dummy Mouse Enhancers Ensembl
Benchmarks: ETTh1 ETTm1 Weather Electricity Traffic Dummy Mouse Enhancers Ensembl
Metrics: MSE inference acceleration (speed-up) accuracy FLOPs
Baselines: Transformer Autoformer FEDformer Informer Non-stationary Transformer Global token merging Local pruning Gaussian filtering
Builds on: Token Merging (ToMe) Informer Autoformer FEDformer Hyena
Closest papers by meaning
- Video Token Merging for Long Video Understanding (NeurIPS 2024 · similarity 0.88)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.86)
- Accelerating Transformers with Spectrum-Preserving Token Merging (NeurIPS 2024 · similarity 0.86)
- Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- ToMA: Token Merge with Attention for Diffusion Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.