Research map / Video / Video segmentation and tracking
UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniSTD, a unified ViT-based framework that initializes from CLIP, uses rank-adaptive MoE LoRA and a lightweight temporal attention module to jointly learn 10 spatiotemporal prediction tasks across 4 disciplines in one model.
Problem
Spatiotemporal prediction methods rely on task-specific architectures that generalize poorly and conflict when trained jointly across diverse disciplines, raising cost and limiting scalability.
Approach
Use a standard pretrained Transformer (CLIP-initialized ViT) as a shared backbone, adapt it with LoRA-based rank-adaptive mixture-of-experts whose ranks are relaxed to continuous values for gradient optimization, and add a zero-initialized lightweight temporal attention module to capture temporal dynamics.
Key results
- UniSTD reaches 39.6 PSNR / 0.98 SSIM on TaxiBJ, versus 29.7 PSNR for the next-best baseline SimVP, while jointly handling 10 tasks in one model.
- On BAIR, UniSTD attains 20.3 PSNR vs 16.3 for Earthformer; on Human3.6M it attains 33.2 PSNR vs 23.7 for the best 3-task baseline.
- Ablation on 4 tasks shows removing both rank-adaptive MoE and temporal attention lowers Cityscapes PSNR from 27.4 to 26.2 and KTH from 28.2 to 27.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the unified UniSTD model instead of task-specific baselines | spatiotemporal prediction models · TaxiBJ | PSNR: 39.6 vs. 39.3 (-0.3) | Helped |
| Use the unified UniSTD model instead of task-specific baselines | spatiotemporal prediction models · Cityscapes | PSNR: 27.4 vs. 26.7 (-0.7) | Helped |
| Add temporal attention (ablation) | UniSTD · KTH Action | PSNR: 27.7 (-0.5) -> 28.2 | Helped |
| Add temporal attention (ablation) | UniSTD · Cityscapes | PSNR: 26.7 (-0.7) -> 27.4 | Helped |
| Add temporal attention (ablation) | UniSTD · BAIR | PSNR: 20.1 (-0.1) -> 20.2 | Helped |
| Add temporal attention (ablation) | UniSTD · TaxiBJ | PSNR: 39.4 (-0.1) -> 39.5 | Helped |
| Add rank-adaptive MoE while retaining temporal attention (ablation) | UniSTD · KTH Action | PSNR: 27.3 (-0.9) -> 28.2 | Helped |
| Add rank-adaptive MoE while retaining temporal attention (ablation) | UniSTD · Cityscapes | PSNR: 26.2 (-1.2) -> 27.4 | Helped |
| Replace random initialization with CLIP initialization (ablation) | UniSTD · KTH Action | PSNR: 27.1 -> 28.2 | Helped |
| Replace random initialization with CLIP initialization (ablation) | UniSTD · Cityscapes | PSNR: 26.7 -> 27.4 | Helped |
| Replace random initialization with CLIP initialization (ablation) | UniSTD · BAIR | PSNR: 20.1 -> 20.2 | No clear effect |
| Use CLIP instead of LAION initialization (ablation) | UniSTD · KTH Action | PSNR: 28.1 -> 28.2 | No clear effect |
Surprising findings
- Existing single-architecture methods degrade sharply when trained on only 3 jointly-trained tasks, while UniSTD scales to 10 tasks without reported loss.
Limitations
- Results of baselines are often from single-task or smaller joint settings, making comparisons hard to interpret.
- Experiments with a fixed ViT-Base backbone; scaling behavior beyond 10 tasks not explored.
Experimental setup
Datasets: TaxiBJ Traffic4Cast Moving MNIST BAIR Human3.6M KTH Action Cityscapes KITTI SEVIR ENSO
Benchmarks: TaxiBJ Traffic4Cast Moving MNIST BAIR Human3.6M KTH Action Cityscapes KITTI SEVIR ENSO
Metrics: PSNR SSIM CSI NINO
Baselines: TAU SimVP SimVPv2 Earthformer MCVD
Builds on: LoRA OpenSTL PredBench SimVP Earthformer
Closest papers by meaning
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning (CVPR 2021 · similarity 0.87)
- Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations (AAAI 2024 · similarity 0.86)
- Get Rid of Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework (NeurIPS 2024 · similarity 0.85)
- Diffusion Transformers as Open-World Spatiotemporal Foundation Models (NeurIPS 2025 · similarity 0.85)
- MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence (ICML 2026 · similarity 0.85)
- SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.