Research map / Video / Video segmentation and tracking
DFDNet: Disentangling and Filtering Dynamics for Enhanced Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DFDNet, a non-recurrent video predictor that disentangles spatiotemporal features into horizontal and vertical dynamic sequences and filters transient high-frequency low-amplitude components with a learnable FFT threshold filter.
Problem
Video prediction models entangle dynamics across spatial directions and are disturbed by transient high-dynamic information such as noise and flicker, which hampers dynamic extraction.
Approach
Decompose spatially encoded video features along height and width by folding those axes into the time/channel dimension so that each MLP sees one direction's dynamics, and apply a learnable FFT amplitude threshold to remove high-frequency low-amplitude components before temporal MLP prediction, with an STAU unit to recover cross-direction correlations.
Key results
- On Moving MNIST (10→10), DFDNet reaches MSE 13.9 vs 17.3 for prior best SIAM, a 19.65% MSE improvement.
- On SJTU4K, DFDNet improves LPIPS to 51.41 at t=5 and 61.93 at t=8 over STRPM (57.12 and 66.68) with 21.12s inference.
- On KTH (10→20), DFDNet achieves LPIPS 12.5 and PSNR 35.11 vs TAU's 21.2 and 34.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DFDNet instead of SIAM | Video prediction · Moving MNIST (10→10 frames) | MSE: 17.3 -> 13.9 | Helped |
| Use DFDNet instead of TAU | Video prediction · KTH (10→20 frames) | PSNR: 34.13 -> 35.11 | Helped |
| Use DFDNet instead of TAU | Video prediction · KTH (10→40 frames) | PSNR: 33.01 -> 33.68 | Helped |
| Use DFDNet instead of STRPM | Video prediction · Human3.6M, t=5 | LPIPS: 9.74 -> 7.32 | Helped |
| Use DFDNet instead of STRPM | Video prediction · SJTU4K, t=8 | LPIPS: 66.68 -> 61.93 | Helped |
| Use DFDNet instead of STDiff | Video prediction on KTH (10→20 frames) · KTH (10→20 frames) | FVD: 89.67 -> 94.82 | Hurt |
| Use DFDNet instead of STDiff | Video prediction on KTH (10→20 frames) · KTH (10→20 frames) | inference time: 6.56s -> 0.44s | Helped |
| Add STAU (ablation) | DFDNet · Moving MNIST (10→10 frames) | MSE: 15.3 -> 13.9 | Helped |
| Add Feature Decomposition on the H and W branches (ablation) | DFDNet · Moving MNIST (10→10 frames) | MSE: 41.6 -> 13.9 | Helped |
| Add Learnable Threshold Filters (ablation) | DFDNet · Moving MNIST (10→10 frames) | MSE: 14.2 -> 13.9 | Helped |
Surprising findings
- Compared with the diffusion model STDiff on KTH, DFDNet has slightly worse FVD (94.82 vs 89.67) but is about 15x faster (0.44s vs 6.56s).
Limitations
- Ablation of the filter and decomposition is only reported on Moving MNIST.
- Slightly higher FVD than the diffusion baseline STDiff on KTH, so generation quality trails diffusion.
- Inference time is higher than SimVP (43.06 ms vs 9.89 ms).
Experimental setup
Datasets: Moving MNIST KTH Human3.6M SJTU4K
Benchmarks: Moving MNIST KTH Human3.6M SJTU4K
Metrics: MSE MAE SSIM PSNR LPIPS FVD inference time
Baselines: ConvLSTM PredRNN PredRNNv2 E3D-LSTM PhyDNet SimVP CrevNet TAU FFINet SwinLSTM MIMO-VP SIAM
Builds on: SimVP TAU SwinLSTM
Closest papers by meaning
- Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement (CVPR 2024 · similarity 0.87)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.86)
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.85)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.85)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.85)
- Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.