Research map / Video / Video segmentation and tracking
Motion Graph Unleashed: A Novel Approach to Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the motion graph, which represents image patches as interconnected graph nodes to predict per-pixel multi-flow motion for short-term video prediction, matching or beating SOTA on UCF Sports, KITTI and Cityscapes with much smaller models and less GPU memory.
Problem
Short-term high-resolution video prediction needs motion representations that capture complex motion patterns (blur, deformation, perspective) without excessive memory cost; optical flow is single-valued and motion matrices are O(n^2).
Approach
Build a multi-view graph over downsampled image patches with spatial, forward and backward edges (top-K cosine-similarity neighbors), initialize nodes with tendency and location features, run graph message passing, then decode k weighted dynamic vectors per pixel and warp frames with multi-flow forward splatting.
Key results
- On UCF Sports MMVP split (full set), the method reaches SSIM 0.9314 and PSNR 30.49 versus MMVP's 0.9300 and 30.35, with 0.60M parameters vs 2.79M (about 78% smaller) and 2.38GB vs 4.53GB GPU memory (about 47% lower).
- On UCF Sports STRPM split at t=10, PSNR 23.27 and LPIPS×100 19.94, improving on MMVP's 23.25 and 22.24.
- On Cityscapes t+1 the method reports LPIPS×100 of 4.13 versus DMVFN's 5.58, while KITTI t+1 MS-SSIM is 87.70 versus DMVFN's 88.53.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Motion Graph instead of MMVP | UCF Sports STRPM split · UCF Sports STRPM split, t=5 | LPIPS × 100: 7.88 -> 6.61 | Helped |
| use Motion Graph instead of MMVP | UCF Sports MMVP split · UCF Sports MMVP split, full set | LPIPS: 0.1062 -> 0.0823 | Helped |
| use Motion Graph instead of DMVFN | Cityscapes · Cityscapes, t+1 | LPIPS × 10^-2: 5.58 -> 4.13 | Helped |
| use Motion Graph instead of DMVFN | KITTI · KITTI, t+1 | LPIPS × 10^-2: 10.74 -> 9.50 | Helped |
| increase k from 1 to 10 (ablation) | Motion Graph · UCF Sports MMVP split, full set | SSIM: 0.9199 -> 0.9314 | Helped |
| increase graph views from 1 to 4 (ablation) | Motion Graph · UCF Sports MMVP split, full set | SSIM: 0.9058 -> 0.9314 | Helped |
| add the location feature to node initialization (ablation) | Motion Graph · UCF Sports MMVP split, full set | SSIM: 0.9247 -> 0.9314 | Helped |
| allow 10 predicted dynamic vectors instead of 1 (ablation) | Motion Graph with k=1 · UCF Sports MMVP split, full set, 100 epochs | SSIM: 0.8742 -> 0.9199 | Helped |
| increase embedded motion modes from k=1 to k=10 (ablation) | Motion Graph with 1 predicted vector · UCF Sports MMVP split, full set, 100 epochs | SSIM: 0.8742 -> 0.9212 | Helped |
| add spatial edges (ablation) | Motion Graph without spatial or backward edges · KITTI, t+1 | PSNR: 21.55 -> 21.64 | Helped |
| add backward edges (ablation) | Motion Graph with spatial edges · KITTI, t+1 | PSNR: 21.64 -> 21.71 | Helped |
Surprising findings
- The authors state the method is slower at inference than DMVFN, so efficiency gains are limited to model size and GPU memory.
- On Cityscapes and KITTI at t+1 and t+3 the method does not beat DMVFN on MS-SSIM, so the claim of matching SOTA is mixed.
Limitations
- No significant improvement in inference speed; slower than DMVFN.
- Non-generative model struggles with occlusions requiring generation of unseen objects.
- Fails on sudden or unpredictable motions such as kicking and diving in UCF Sports.
Experimental setup
Datasets: UCF Sports KITTI Cityscapes
Benchmarks: UCF Sports STRPM split UCF Sports MMVP split KITTI Cityscapes
Metrics: PSNR LPIPS SSIM MS-SSIM model size GPU memory
Baselines: ConvLSTM PredRNN PredRNN++ E3D-LSTM CrevNet STRPM STIP DMVFN MMVP OPT SimVP
Builds on: MMVP DMVFN STRPM SimVP
Closest papers by meaning
- GCNext: Towards the Unity of Graph Convolutions for Human Motion Prediction (AAAI 2024 · similarity 0.87)
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.85)
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.85)
- MotionRAG: Motion Retrieval-Augmented Image-to-Video Generation (NeurIPS 2025 · similarity 0.85)
- DFDNet: Disentangling and Filtering Dynamics for Enhanced Video Prediction (AAAI 2025 · similarity 0.85)
- TTVidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.