Research map / Video / Video generation
Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Continuous Video Process (CVP), which models consecutive video frames as endpoints of a continuous interpolation process with a zero-noise-at-endpoints schedule and a diffusion-like variational objective, for video prediction on KTH, BAIR, Human3.6M and UCF101.
Problem
Video prediction with diffusion models usually treats videos as independent images and relies on external temporal constraints such as temporal attention, which is costly; sampling also requires many steps.
Approach
Treat the transition between consecutive frames as a continuous interpolation process x_t=(1-t)x+ty+noise with g(t)=-t log t, which vanishes at both endpoints, and learn the reverse process with a simplified loss that predicts the endpoint y; no temporal attention is used.
Key results
- On KTH (64x64, 10 context frames, 30 predicted), CVP reaches FVD 140.6 vs 222 for SRVP and 180 for RIVER.
- On BAIR with 1 context frame predicting 15 frames, CVP achieves FVD 70.1, versus 66.9 for VDM which is the best reported there.
- On UCF101 (5 context, 16 predicted), CVP achieves FVD 245.2 vs 356 for RaMViD, a new state of the art as claimed.
- The authors report 75% fewer sampling steps than a diffusion baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CVP instead of RIVER | KTH video prediction · KTH, 30 predicted frames | FVD: 180 -> 140.6 | Helped |
| use CVP instead of Grid-keypoints | KTH video prediction · KTH, 40 predicted frames | FVD: 144.2 -> 120.1 | Helped |
| use CVP instead of VDM | BAIR video prediction · BAIR, 1 context frame, 15 predicted frames | FVD: 66.9 -> 70.1 | Hurt |
| use CVP instead of MCVD | BAIR video prediction · BAIR, 2 context frames, 14 predicted frames | FVD: 87.9 -> 68.2 | Helped |
| use CVP instead of MCVD | BAIR video prediction · BAIR, 2 context frames, 28 predicted frames | FVD: 118.4 -> 95.1 | Helped |
| use CVP instead of Grid keypoint | Human3.6M video prediction · Human3.6M, 30 predicted frames | FVD: 166.1 -> 144.5 | Helped |
| use CVP instead of RaMViD | UCF101 video prediction · UCF101, 16 predicted frames | FVD: 356 -> 245.2 | Helped |
| predict 1 frame at a time instead of 2 (ablation) | CVP · BAIR, 2 context frames, 14 predicted frames | FVD: 68.2 -> 65.1 | Helped |
| predict 1 frame at a time instead of 2 (ablation) | CVP · BAIR, 2 context frames, 28 predicted frames | FVD: 95.1 -> 85.1 | Helped |
Surprising findings
- On BAIR with one context frame, CVP (FVD 70.1) does not beat VDM (FVD 66.9), contrary to the paper's general state-of-the-art claim.
Limitations
- Relies on a limited fixed context window (e.g. 4 frames) and cannot exploit longer history.
- Requires multiple sampling steps per frame, which becomes a bottleneck for long predictions.
- Experiments limited by compute (two A6000 GPUs), so scaling was not explored.
Experimental setup
Datasets: KTH BAIR robot pushing Human3.6M UCF101
Benchmarks: KTH BAIR Human3.6M UCF101
Metrics: FVD PSNR SSIM
Baselines: SVG-LP SAVP MCVD SLAMP SRVP RIVER Struct-vRNN Grid-keypoints DVG CCVS RaMViD FitVid
Builds on: denoising diffusion probabilistic models InDI Cold diffusion MCVD
Closest papers by meaning
- STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · similarity 0.88)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.88)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.88)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.88)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.87)
- TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.