Research map / Video / Video segmentation and tracking
Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WaST, a wavelet-based recurrent-free spatiotemporal predictor using 3D discrete wavelet transform, a Time-Frequency Aware Translator, and a wavelet-domain High-Frequency Focal Loss to predict future frames efficiently.
Problem
Spatiotemporal predictive learning (future frame prediction) with recurrent units is computationally expensive and performs poorly on real-world scenes, while recurrent-free models neglect time-frequency dependencies and lose prediction details.
Approach
Decompose sequences with multi-level 3D Haar wavelet transform into low- and high-frequency subbands, model intra-frame frequency and inter-frame temporal variation separately in a Time-Frequency Aware Translator, and supervise high-frequency content with a wavelet-domain focal loss added to MSE.
Key results
- On Kitti&Caltech, WaST reaches MSE 123.5 and SSIM 0.9468 with 29.4 GFLOPs, versus MIM at 1858.0 GFLOPs.
- On Human3.6M, WaST achieves SSIM 0.9839 and PSNR 34.19, a 21x FLOPs reduction over PredRNN++ (49.3G vs 1033.0G).
- On WeatherBench temperature forecasting, WaST reduces MSE from 1.331 (PredRNN) to 1.098 with 1.5G FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WaST instead of MIM | Kitti&Caltech prediction · Kitti&Caltech | MSE: 125.1 -> 123.5 | Helped |
| use WaST instead of PredRNN++ | Human3.6M prediction · Human3.6M | MSE: 110.0 -> 109.8 | Helped |
| use WaST instead of TAU | TaxiBJ prediction · TaxiBJ | MSE × 100: 34.4 -> 30.8 | Helped |
| use WaST instead of TAU | WeatherBench temperature forecasting · WeatherBench temperature | MSE: 1.224 -> 1.098 | Helped |
| add HFFL (ablation) | WaST · WeatherBench humidity | MAE: 3.721 -> 3.694 | Helped |
| add HFFL (ablation) | WaST · WeatherBench cloud cover | MAE: 0.1469 -> 0.1452 | Helped |
| add HFFL (ablation) | WaST · WeatherBench humidity | RMSE: 5.600 -> 5.569 | Helped |
| add HFFL (ablation) | WaST · WeatherBench cloud cover | RMSE: 0.2167 -> 0.2150 | Helped |
| replace TF-Aware Translator with ViT (ablation) | WaST · WeatherBench humidity | MAE: 3.694 -> 3.911 | Hurt |
| replace TF-Aware Translator with Uniformer (ablation) | WaST · WeatherBench humidity | MAE: 3.694 -> 3.914 | Hurt |
| replace TF-Aware Translator with MLP-Mixer (ablation) | WaST · WeatherBench humidity | MAE: 3.694 -> 3.950 | Hurt |
| replace TF-Aware Translator with ConvMixer (ablation) | WaST · WeatherBench humidity | MAE: 3.694 -> 3.909 | Hurt |
Surprising findings
- Recurrent-free architectures outperformed recurrent-based ones in the humidity and cloud cover ablation.
- An oversized 51x51 convolution kernel on 32x32 TaxiBJ inputs was used to capture low-frequency features, yielding near-perfect predictions.
Limitations
- Relies on Haar wavelets only; other wavelet families are not explored.
- Evaluated on video and climate datasets with fixed frame counts; generalization to other domains is not shown beyond the listed datasets.
Experimental setup
Datasets: KITTI Caltech Pedestrian Human3.6M TaxiBJ WeatherBench
Benchmarks: Kitti&Caltech Human3.6M TaxiBJ WeatherBench
Metrics: MSE MAE RMSE SSIM PSNR FLOPs parameters
Baselines: ConvLSTM PredRNN PredRNN++ MIM E3D-LSTM PhyDNet MAU PredRNNv2 SimVP TAU PredNet
Builds on: SimVP Focal Frequency Loss MetaFormer TAU
Closest papers by meaning
- PFGNet: A Fully Convolutional Frequency-Guided Peripheral Gating Network for Efficient Spatiotemporal Predictive Learning (CVPR 2026 · similarity 0.89)
- Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning (CVPR 2023 · similarity 0.89)
- UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines (CVPR 2025 · similarity 0.86)
- Learning Spatiotemporal Dynamical Systems from Point Process Observations (ICLR 2025 · similarity 0.86)
- SCENT: Robust Spatiotemporal Learning for Continuous Scientific Data via Scalable Conditioned Neural Fields (ICML 2025 · similarity 0.85)
- Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.