Research map / Time series / Foundation models for time series
Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WaveToken, a wavelet-based tokenizer (scale, DWT, threshold, quantize into 1024 tokens) that pre-trains a T5 encoder-decoder to autoregressively forecast wavelet coefficients, evaluated on 42 in-domain and zero-shot datasets.
Problem
Choosing an effective discrete vocabulary for real-valued time series so that LLM-style autoregressive transformers can forecast compactly and generalize to unseen datasets.
Approach
Tokenize time series by applying a maximally decimated discrete wavelet transform after z-score scaling, then quantize the coefficients with a shared 1024-bin vocabulary and concatenate coarse-to-fine coefficient groups so a T5 model forecasts them autoregressively.
Key results
- WaveToken with 1024-token vocabulary outperforms Chronos on 75% of model-size/metric comparisons in-domain and achieves 83% success rate against same-size Chronos models zero-shot.
- WaveToken achieves the best average rank across all datasets on Benchmark I and II for the three metrics, per the paper's claims.
- Using a 1024-token vocabulary versus Chronos's 4096 gives a vocabulary one quarter the size while reaching comparable or better WQL/MASE on in-domain data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WaveToken instead of other foundation models | Time-series forecasting models · Benchmark II (zero-shot) | WQL | Helped |
| Use WaveToken instead of task-specific models | Time-series forecasting models · Benchmark II (zero-shot) | VRSE | Helped |
| Use WaveToken instead of other foundation models | Time-series forecasting models · Long-horizon zero-shot benchmark, H × 3 | WQL | Hurt |
| Increase vocabulary size up to 1024 tokens (ablation) | WaveToken-Small · In-domain and zero-shot benchmarks | Forecasting accuracy | Helped |
| Increase vocabulary size beyond 1024 tokens (ablation) | WaveToken-Small · In-domain and zero-shot benchmarks | WQL | No clear effect |
| Use Biorthogonal-2.2 instead of other wavelet families (ablation) | WaveToken-Small · Forecasting benchmarks | Forecasting accuracy | Helped |
| Use wavelet families with higher vanishing moments (ablation) | WaveToken-Small · Forecasting benchmarks | Forecasting accuracy | Hurt |
| Increase decomposition depth beyond the first level (ablation) | WaveToken-Small · Forecasting benchmarks | Forecasting accuracy | Hurt |
| Use VisuShrink thresholding instead of no thresholding (ablation) | WaveToken · Single-GPU hyperparameter experiments | Forecasting accuracy | Helped |
| Use no thresholding instead of VisuShrink (ablation) | WaveToken · Eight-GPU training | Generalization | Helped |
Surprising findings
- VisuShrink thresholding was best in the single-GPU ablation but reversed to no-thresholding in the 8-GPU final setting.
- Single-level decomposition was sufficient; deeper levels made attention harder to focus on relevant coefficient groups.
Limitations
- Autoregressive decoding over wavelet coefficients is slower at inference than patch-based models.
- Context length limited to 512 in the current tokenizer.
- Thresholding choices depended on GPU count, suggesting sensitivity to training setup.
Experimental setup
Datasets: Chronos benchmark training corpus (13 pre-training-only datasets) Benchmark I (15 in-domain datasets) Benchmark II (27 zero-shot datasets) TSMixup synthetic Gaussian process synthetic
Benchmarks: Benchmark I (in-domain, 15 datasets) Benchmark II (zero-shot, 27 datasets) long-horizon variant of Benchmark II
Metrics: WQL MASE VRSE average rank aggregate relative score
Baselines: Chronos-Mini Chronos-Small Chronos-Base Chronos-Large TimesFM Moirai-Base Moirai-Large Lag-Llama PatchTST DeepAR TFT Seasonal Naive
Builds on: Chronos T5 VisuShrink FDRC
Closest papers by meaning
- WaveletMixer: A Multi-Resolution Wavelets Based MLP-Mixer for Multivariate Long-Term Time Series Forecasting (AAAI 2025 · similarity 0.90)
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models (ICLR 2024 · similarity 0.89)
- AutoTimes: Autoregressive Time Series Forecasters via Large Language Models (NeurIPS 2024 · similarity 0.89)
- Large Pre-trained time series models for cross-domain Time series analysis tasks (NeurIPS 2024 · similarity 0.89)
- TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting (NeurIPS 2025 · similarity 0.89)
- Towards a General Time Series Forecasting Model with Unified Representation and Adaptive Transfer (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.