Research map / Time series / Long-term forecasting
VarDrop: Enhancing Training Efficiency by Reducing Variate Redundancy in Periodic Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VarDrop, which drops redundant variate tokens per batch during training by grouping variates with k-dominant frequency hashing and stratified sampling, cutting attention cost of variate-tokenized Transformers (iTransformer).
Problem
Variate tokenization in multivariate time series forecasting incurs quadratic self-attention cost in the number of variates, limiting training efficiency on high-dimensional datasets, which contain many redundant, highly correlated variates.
Approach
Hash each variate by the ordered indices of its top-k FFT dominant frequencies (batch-averaged, low-pass filtered), group variates with identical hashes, and randomly sample a bounded number per group so that only representative variate tokens enter scaled dot-product attention during training only.
Key results
- On Electricity, VarDrop (on iTransformer backbone) averages MSE 0.181 / MAE 0.271, within about 0.6% / 0.4% relative error of the full-token ground truth and better than efficient baselines such as Flowformer and Informer.
- VarDrop reduces tokens by 63.33% (Electricity), 78.14% (Traffic), 66.19% (Weather) and 85.38% (Solar-Energy).
- Training time per iteration on iTransformer drops from 68 ms to 33 ms (2.06x), and GPU memory on Electricity is 2.22 GB versus 3.41 GB for iTransformer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply VarDrop variate reduction instead of the efficient baseline's token handling | Variate-tokenized Transformers · Electricity, Traffic, Weather, and Solar-Energy | forecasting error (MSE and MAE): Overall MSE: 0.299 -> 0.277; overall MAE: 0.292 -> 0.275 | Helped |
| Apply VarDrop to iTransformer training | iTransformer · Electricity, Traffic, Weather, and Solar-Energy (input-96-predict-96) | training time (ms/iteration): 68.0 -> 33.0 | Helped |
| Apply VarDrop token reduction | iTransformer · Electricity (input-96-predict-96) | GPU memory footprint: 3.41\,GB -> 2.22\,GB | Helped |
| Increase the dominant-frequency hash length k (ablation) | VarDrop · Electricity | MSE | Helped |
| Increase the dominant-frequency hash length k (ablation) | VarDrop · Electricity | MAE | Helped |
| Vary maximum group size gs (Figure sensitivity sweep) (ablation) | VarDrop · Electricity | MSE | Mixed |
| Vary maximum group size gs (Figure sensitivity sweep) (ablation) | VarDrop · Electricity | MAE | Mixed |
| Vary maximum group size gs (ablation) | VarDrop · Electricity | token reduction ratio: 93.1%, 76.1%, 63.5%, 47.3% | Hurt |
Surprising findings
- Higher token reduction does not necessarily hurt accuracy; certain k and group-size settings outperform others with more reduction, which the authors attribute to k-DFH grouping precision.
Limitations
- Hyperparameters k and gs and cutoff epsilon were chosen without sophisticated tuning, and the authors leave optimization to future work.
- Efficiency gains are evaluated only on periodic benchmark datasets, and the method is designed for periodic time series.
- Many baseline numbers in the main table are taken from prior work rather than re-run.
Experimental setup
Datasets: Electricity Traffic Weather Solar-Energy
Benchmarks: Electricity Traffic Weather Solar-Energy
Metrics: MSE MAE training time per iteration (ms) GPU memory footprint token reduction ratio
Baselines: FlashAttention Flowformer Reformer Informer iTransformer
Builds on: iTransformer Dropout FlashAttention Informer Reformer Flowformer
Closest papers by meaning
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.87)
- Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting (NeurIPS 2024 · similarity 0.87)
- Revisiting Attention for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.86)
- HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting (AAAI 2025 · similarity 0.86)
- Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences (AAAI 2025 · similarity 0.86)
- Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.