Research map / Time series / Long-term forecasting
TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TimeDART, a self-supervised time series pre-training framework that combines a causal Transformer encoder with patch-level denoising diffusion decoding under an autoregressive objective, evaluated on forecasting and classification.
Problem
Autoregressive self-supervised pre-training for time series struggles to capture fine-grained local patterns and, with squared-error loss, implicitly assumes a Gaussian predictive distribution that fits real-world series poorly.
Approach
Combine a causal Transformer encoder (left-to-right patch modeling) with a denoising diffusion decoder that reconstructs independently noised patches conditioned on encoder outputs, replacing MSE with a diffusion (ELBO-derived) loss within the autoregressive objective.
Key results
- TimeDART achieves best results in 83.3% of 24 forecasting metrics (in-domain) and reduces MSE by 6.8% vs random initialization and 3% vs the best self-supervised baseline.
- On HAR, Epilepsy and EEG classification, TimeDART reaches accuracy 0.9247, 0.9712 and 0.8269 respectively, with ~5.7% average accuracy gain over random initialization.
- Removing both autoregressive and diffusion components (w/o AR-Diff) degrades ETTh2, ETTm2, Electricity and PEMS04 results; removing AR alone makes performance worse than random init on several datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the autoregressive mechanism (ablation) | TimeDART · ETTm2 | MSE: 0.281 -> 0.257 | Helped |
| Add the autoregressive mechanism (ablation) | TimeDART · HAR | Accuracy: 0.8966 -> 0.9247 | Helped |
| Add the denoising diffusion process (ablation) | TimeDART · Electricity | MSE: 0.164 -> 0.163 | Helped |
| Add the denoising diffusion process (ablation) | TimeDART · Epilepsy | Accuracy: 0.9598 -> 0.9712 | Helped |
| Add both autoregressive mechanism and diffusion (ablation) | TimeDART · PEMS04 | MSE: 0.149 -> 0.133 | Helped |
| Use cosine instead of linear noise scheduler (ablation) | TimeDART · PEMS04 | MSE: 0.143 -> 0.133 | Helped |
| Vary diffusion steps between 750, 1000, and 1250 (ablation) | TimeDART · ETTm2 | MSE: 0.258; 0.257; 0.257 | No clear effect |
| Replace Transformer encoder with causal TCN (ablation) | TimeDART · ETTm2 | MSE: 0.257 -> 0.263 | Hurt |
| Replace Transformer encoder with causal TCN (ablation) | TimeDART · HAR | Accuracy: 0.9247 -> 0.9252 | No clear effect |
| Use a self-only denoising decoder mask rather than adding extra information (ablation) | TimeDART · Time-series denoising | pre-training performance | Helped |
| Share embedding weights for clean and noise-added patches (ablation) | TimeDART · Time-series pre-training | performance | Helped |
| Increase look-back window length (ablation) | TimeDART · Time-series forecasting | MSE | Helped |
Surprising findings
- Removing the autoregressive mechanism yields performance even worse than random initialization in the ablation.
- Cosine noise scheduler performs substantially better than linear; linear sometimes underperforms random initialization.
Limitations
- Forecasting with a vanilla Transformer encoder is weaker with TCN backbone, attributed to mismatch with the Transformer decoder used in pre-training.
- Cross-domain pre-training underperforms in-domain on the Weather dataset, attributed to shared model-specific parameters across domains.
- Patch length and decoder depth require dataset-specific tuning; smaller patches increase computational cost.
Experimental setup
Datasets: ETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic Weather Exchange PEMS03 PEMS04 PEMS07 PEMS08
Benchmarks: ETTh1 ETTh2 ETTm1 ETTm2 Electricity Traffic Weather Exchange PEMS03 PEMS04 PEMS07 PEMS08
Metrics: MSE MAE Accuracy Macro-F1
Baselines: SimMTM PatchTST TimeMAE CoST DLinear FormerTime Random Init.
Builds on: DDPM PatchTST TimeGrad MSE-based autoregressive pre-training
Closest papers by meaning
- Latent Diffusion Transformer for Probabilistic Time Series Forecasting (AAAI 2024 · similarity 0.91)
- Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series (NeurIPS 2024 · similarity 0.89)
- Diffusion-TS: Interpretable Diffusion for General Time Series Generation (ICLR 2024 · similarity 0.89)
- Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approach (NeurIPS 2025 · similarity 0.88)
- TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (ICLR 2024 · similarity 0.88)
- Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.