Research map / Video / Video generation
Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theory paper showing diffusion transformers can approximate the score of Gaussian process data by unrolling gradient descent in attention layers, giving a sample complexity bound that depends on spatial-temporal dependency decay, with supporting synthetic and semi-synthetic experiments.
Problem
Whether diffusion transformers efficiently capture spatial-temporal dependencies in sequential data, and how those dependencies affect learning efficiency, is theoretically unresolved.
Approach
Represent the Gaussian process score as the minimizer of a strongly convex quadratic, approximate it by truncated gradient descent (dropping correlations beyond length J), and show a transformer can unroll these gradient steps, with attention computing the temporal correlation from time embeddings.
Key results
- Relative covariance error decreases with sample size; larger nu or smaller ell gives better performance, consistent with the generalization bound.
- Sample complexity bound scales as sqrt(ell^{1/nu} kappa^2 N d^3 / n), i.e. order 1/sqrt(n), with weak dependence on N for fast-decaying covariance.
- In the 5th block, query-key interaction is dominated by the time-embedding sub-block, and attention score matrices match the ground-truth temporal dependency from the 3rd layer onward.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of training sequences | Diffusion Transformer · Synthetic Gaussian process (d=8, N=128) | relative covariance error | Helped |
| Increase covariance decay exponent ν | Diffusion Transformer · Synthetic Gaussian process (ℓ=64) | relative covariance error | Helped |
| Decrease covariance bandwidth ℓ | Diffusion Transformer · Synthetic Gaussian process (ν=2) | relative covariance error | Helped |
| Use time-embedding components in the query-key matrices | Diffusion Transformer · Synthetic Gaussian process | attention weight magnitude | Helped |
| Progressively denoise in the backward diffusion process | Diffusion Transformer · Synthetic Gaussian process | similarity of attention score matrix to ground-truth temporal covariance | Helped |
| Use attention layers starting from the 3rd layer to represent temporal dependencies | Diffusion Transformer · Synthetic Gaussian process | capture of temporal dependencies | Helped |
| Train a latent diffusion transformer on ball-motion sequences | Diffusion Transformer with a 2D VAE · Semi-synthetic video data (240-frame ball-motion videos) | spatial-temporal dependency capture | Helped |
Limitations
- Analysis restricted to Gaussian process data, which is simpler than real video or sequential data.
- Requires the truncated covariance to be PSD and ell <= c^nu for technical convenience.
- Number of transformer blocks scales with the condition number, which may be large for slowly decaying covariances.
Experimental setup
Datasets: synthetic Gaussian process data semi-synthetic bouncing ball video data
Benchmarks: synthetic Gaussian process data
Metrics: relative error of estimated sample covariance matrix total variation bound (theory)
Baselines: U-Net diffusion models
Builds on: DiT score-based diffusion models latent diffusion
Closest papers by meaning
- Generative Pre-trained Autoregressive Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- STDD: Spatio-Temporal Dual Diffusion for Video Generation (CVPR 2025 · similarity 0.88)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series (NeurIPS 2024 · similarity 0.88)
- Dynamic Diffusion Transformer (ICLR 2025 · similarity 0.87)
- Distributional Diffusion Models with Scoring Rules (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.