Research map / Time series / Causal and generative time series
Retrieval-Augmented Diffusion Models for Time Series Forecasting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RATD, a retrieval-augmented time series diffusion model that retrieves embedding-nearest historical windows from a database as references and injects them into the denoiser via Reference Modulated Attention to guide forecasting.
Problem
Existing time series diffusion models for forecasting are unstable on complex cases, lacking guidance and suffering from small, imbalanced datasets.
Approach
Retrieve the k nearest training windows by pre-trained encoder embedding of the history, then use them as references in a CSDI-style transformer denoiser through a Reference Modulated Attention module placed at the front of each residual block, so references guide every denoising step.
Key results
- RATD reaches MSE 0.013 on Exchange, 0.784 on Wind, and 0.281 on Weather, best among compared methods on those datasets, with CRPS 0.673 on Wind.
- On the MIMIC-IV rare-case subset, RATD achieves MSE 0.206 versus 0.423 for the best baseline iTransformer.
- Ablation: the CSDI+RMA variant reduces Electricity MSE from 0.379 (CSDI) to 0.151, while random references degrade performance versus no references.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrieval references instead of using no references (ablation) | RATD · Exchange, Wind, Electricity, Weather | MSE: 0.077 -> 0.013; 1.066 -> 0.784; 0.379 -> 0.151; 0.356 -> 0.281 | Helped |
| Use random references rather than no references (ablation) | RATD · Exchange, Wind, Electricity, Weather | MSE: 0.077 -> 0.153; 1.066 -> 1.593; 0.379 -> 0.471; 0.356 -> 0.431 | Hurt |
| Use embedding-based retrieval rather than DTW retrieval (ablation) | RATD · Exchange, Wind, Electricity, Weather | MSE: 0.075 -> 0.013; 1.073 -> 0.784; 0.357 -> 0.151; 0.361 -> 0.281 | Mixed |
| Use embedding-based retrieval rather than Pearson retrieval (ablation) | RATD · Exchange, Wind, Electricity, Weather | MSE: 0.091 -> 0.013; 1.099 -> 0.784; 0.361 -> 0.151; 0.370 -> 0.281 | Helped |
| Use different embedding encoders (ablation) | RATD · retrieval mechanism | prediction performance | No clear effect |
| Increase retrieval database size n (ablation) | RATD · retrieval database | prediction accuracy | Helped |
| Increase the number of references k (ablation) | RATD · retrieval database | prediction accuracy | No clear effect |
| Add a linear fusion layer for input and references (ablation) | CSDI · Exchange, Electricity, Wind, Weather, Solar, MIMIC-IV | MSE: 0.077 -> 0.075; 0.379 -> 0.316; 1.066 -> 0.932; 0.356 -> 0.349; 0.381 -> 0.369; 0.268 -> 0.265 | Helped |
| Add cross-attention to fuse input and references (ablation) | CSDI · Exchange, Electricity, Wind, Weather, Solar, MIMIC-IV | MSE: 0.077 -> 0.028; 0.379 -> 0.173; 1.066 -> 0.829; 0.356 -> 0.291; 0.381 -> 0.340; 0.268 -> 0.183 | Helped |
| Add RMA rather than basic CSDI (ablation) | CSDI · Exchange, Electricity, Wind, Weather, Solar, MIMIC-IV | MSE: 0.077 -> 0.013; 0.379 -> 0.151; 1.066 -> 0.784; 0.356 -> 0.281; 0.381 -> 0.327; 0.268 -> 0.172 | Helped |
| Predict epsilon rather than x0 (ablation) | RATD · Wind, Weather, Exchange | MSE: 0.784 -> 0.841; 0.281 -> 0.331; 0.013 -> 0.018 | Hurt |
| Place RMA at the front rather than the back (ablation) | RATD · Exchange, Wind, Electricity, Weather | MSE: 0.031 -> 0.013; 0.673 -> 0.784; 0.267 -> 0.161; 0.301 -> 0.281 | Mixed |
Surprising findings
- Random references make performance worse than using no references on several datasets (e.g., Wind MSE 1.593 vs 1.066 without references).
- Predicting x0 outperforms predicting the noise epsilon, with the authors suggesting the reference is more directly related to x0.
Limitations
- Transformer-based design consumes large compute on series with many variables.
- Retrieval preprocessing adds training cost (around ten hours).
- No error bars are reported for experiments.
Experimental setup
Datasets: Electricity Wind Exchange Weather MIMIC-IV-ECG Solar
Benchmarks: Exchange Wind Electricity Weather MIMIC-IV-ECG
Metrics: MSE MAE CRPS
Baselines: CSDI TimeDiff mr-Diff D3VAE iTransformer PatchTST TimesNet FiLM FEDformer FreTS Autoformer Pyraformer
Builds on: CSDI DDPM DiffWave mr-Diff TimeDiff
Closest papers by meaning
- Auto-Regressive Moving Diffusion Models for Time Series Forecasting (AAAI 2025 · similarity 0.90)
- A Non-isotropic Time Series Diffusion Model with Moving Average Transitions (ICML 2025 · similarity 0.89)
- Conditional Diffusion Model with Nonlinear Data Transformation for Time Series Forecasting (ICML 2025 · similarity 0.89)
- Population Aware Diffusion for Time Series Generation (AAAI 2025 · similarity 0.88)
- MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process (ICLR 2024 · similarity 0.88)
- TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.