Research map / Time series / Long-term forecasting
HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HDT, a two-stage hierarchical vector-quantized Transformer that turns high-dimensional multivariate time series forecasting into discrete token generation, conditioning fine-grained target tokens on discrete long-term trend tokens.
Problem
Probabilistic forecasting of high-dimensional multivariate time series over long horizons is hard for existing generative models (diffusion, flows, VAEs), which scale poorly in dimension and are limited to short forecast lengths.
Approach
Two-stage VQGAN-style tokenization of both the target and its moving-average-downsampled trend, with an l2-normalized codebook; stage 2 uses a low-level Transformer to generate trend tokens and a high-level self-conditioned Transformer decoder that attends to them to generate target tokens autoregressively.
Key results
- HDT achieves best CRPS_sum on 16 of the reported dataset/horizon settings in the main table, and reports an average 16.7% CRPS_sum improvement over state-of-the-art baselines.
- On Taxi (pred 48/96/144), HDT reports average CRPS_sum 0.329 vs. 0.360 for MG_TSD and 0.428 for VQ-TR.
- Replacing discrete tokens with continuous representations (C-Transformer) degrades results substantially, e.g., Electricity CRPS_sum 0.327 vs 0.025 at horizon 48.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace discrete token representation with continuous features (ablation) | HDT · Electricity, Traffic, and Taxi (prediction length 48) | CRPS_sum: 0.025(.002) vs 0.327(.004); 0.034(.001) vs 0,467(.011); 0.166(.005) vs 0.861(.006) | Hurt |
| use continuous historical features instead of discrete historical features (ablation) | HDT · Traffic and Taxi (prediction lengths 48, 96, 144) | CRPS_sum: Traffic 48: 0.042(.003) vs 0.034(.001); Taxi 48: 0.206(.004) vs 0.166(.003) | Hurt |
| use discrete context features without the stage-2 encoder (ablation) | HDT · Traffic and Taxi (prediction lengths 48, 96, 144) | CRPS_sum: Traffic 48: 0.038(.002) vs 0.034(.001); Taxi 48: 0.189(.004) vs 0.166(.003) | Mixed |
| concatenate discrete and continuous context features (ablation) | HDT · Traffic and Taxi (prediction lengths 48, 96, 144) | CRPS_sum: Traffic 48: 0.036(.003) vs 0.034(.001); Taxi 48: 0.171(.005) vs 0.166(.003) | Mixed |
| remove the self-condition strategy (ablation) | HDT · long-term forecasting (lengths 96 and 144) | CRPS_sum | Hurt |
| use 3 instead of 4 self-conditioned Transformer layers (ablation) | HDT · Traffic and Taxi (prediction lengths 48, 96, 144) | CRPS_sum: Traffic 96: 0.037(.003); Taxi 48: 0.166(.003); Taxi 96: 0.378(.005) | Mixed |
| increase self-conditioned Transformer layers from 4 to 5 (ablation) | HDT · Traffic and Taxi (prediction lengths 48, 96, 144) | CRPS_sum: Traffic 96: 0.363(.003); Taxi 96: 0.363(.004) | Mixed |
| use HDT instead of MG_TSD | multivariate forecasting · Traffic Avg | CRPS_sum: 0.045 -> 0.039 | Helped |
| use HDT instead of MG_TSD | multivariate forecasting · Taxi Avg | NRMSE_sum: 0.519 -> 0.438 | Helped |
Surprising findings
- Continuous-latent variant (C-Transformer) performs far worse than the discrete variant, which the authors attribute to the unbounded continuous latent space.
Limitations
- Relies on a two-stage pipeline with separately trained codebooks, so training is more complex than end-to-end methods.
- Probabilistic results are mixed on some settings; VQ-TR and MG_TSD win on a few Wikipedia and Solar entries.
Experimental setup
Datasets: Solar Electricity Traffic Taxi Wikipedia COVID Deaths Hospital
Benchmarks: Solar Electricity Traffic Taxi Wikipedia Hospital COVID Deaths
Metrics: CRPS_sum NRMSE_sum QICE PICP
Baselines: VQ-TR MG_TSD TSDiff TimeDiff SSSD D3VAE CSDI TimeGrad Transformer-MAF DeepAR GP-Copula
Builds on: VQGAN VQ-VAE Autoformer series decomposition TimeGrad VQ-TR
Closest papers by meaning
- Considering Nonstationary within Multivariate Time Series with Variational Hierarchical Transformer for Forecasting (AAAI 2024 · similarity 0.91)
- Latent Diffusion Transformer for Probabilistic Time Series Forecasting (AAAI 2024 · similarity 0.90)
- DeformableTST: Transformer for Time Series Forecasting without Over-reliance on Patching (NeurIPS 2024 · similarity 0.89)
- A Closer Look at Transformers for Time Series Forecasting: Understanding Why They Work and Where They Struggle (ICML 2025 · similarity 0.89)
- SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention (ICML 2024 · similarity 0.88)
- Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.