Research map / Video / Video segmentation and tracking
Stochastic video prediction
Predicts future frames with learned dynamics, hierarchical models and example guidance.
125 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video segmentation and tracking, keyword cluster “prediction · person · pose”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
21 papers in 2023–24 and 28 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 4.8% to 5.2%.
Open questions the data points to
- Papers report SAVP on BAIR (FVD) anywhere from 116.4 to 152: comparisons against it depend on whose number you take.
No longer reported by papers here: Human3.6M (PSNR, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Unsupervised Learning From Narrated Instruction Videos (CVPR 2016 · 295 citations)
- 2017 · End-To-End Learning of Driving Models From Large-Scale Video Datasets (CVPR 2017 · 879 citations)
- 2018 · PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning (ICML 2018 · 337 citations)
- 2019 · Animating Arbitrary Objects via Deep Motion Transfer (CVPR 2019 · 373 citations)
- 2020 · BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning (CVPR 2020 · 2,481 citations)
- 2021 · MotionRNN: A Flexible Model for Video Prediction With Spacetime-Varying Motions (CVPR 2021 · 205 citations)
- 2022 · SimVP: Simpler Yet Better Video Prediction (CVPR 2022 · 393 citations)
- 2023 · A Dynamic Multi-Scale Voxel Flow Network for Video Prediction (CVPR 2023 · 91 citations)
- 2024 · Generalized Predictive Model for Autonomous Driving (CVPR 2024 · 47 citations)
- 2025 · Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation (AAAI 2025 · 2 citations)
- 2026 · Identifiable Token Correspondence for World Models (ICML 2026 · 0 citations)
Earliest papers here
- Unsupervised Learning From Narrated Instruction Videos (CVPR 2016 · 295 citations)
- SoundNet: Learning Sound Representations from Unlabeled Video (NeurIPS 2016 · 232 citations)
- Slicing Convolutional Neural Network for Crowd Video Understanding (CVPR 2016 · 87 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Human3.6M · PSNR (higher is better)
6 papers, 2017–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 38.9 | Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (ICLR 2017 · 412 citations) |
| 2018 | 39.7 | Structure Preserving Video Prediction (CVPR 2018 · 49 citations) |
What has been tried
107 papers here report 1,121 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 7%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add deep interaction between temporal and spatial blocks | OpenDV-YouTube · FVD | 244.44 -> 201.69 | Helped | Generalized Predictive Model for Autonomous Driving (CVPR 2024 · 47 citations) |
| add HFFL | WeatherBench humidity · MAE | 3.721 -> 3.694 | Helped | Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations (AAAI 2024 · 30 citations) |
| add compressed-space motion representation | KTH (u=10, v=40) · PSNR | 23.43 -> 25.39 | Helped | ExtDM: Distribution Extrapolation Diffusion Model for Video Prediction (CVPR 2024 · 27 citations) |
| replace the SDE-based motion predictor with an ODE-based motion predictor | KTH, 10 → 20 · iLPIPS | 25.08 -> 15.60 | Hurt | STDiff: Spatio-Temporal Diffusion for Continuous Stochastic Video Prediction (AAAI 2024 · 15 citations) |
| Use momentum-free optimizer (RMSprop) instead of AdamW | Ego4D-stream and ScanNet-stream segmentation · mean IoU / L2 (averaged sweep) | Helped | Learning from One Continuous Video Stream (CVPR 2024 · 8 citations) | |
| Use a parametric interpolator instead of duplication | MiniUCF, Nsyn=Nreal=2 · accuracy (%) | 18.0 ± 0.4 -> 18.6 ± 0.8 | Helped | Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement (CVPR 2024 · 6 citations) |
| add shuffle-based contrastive learning | OpenPVSG, vIoU 0.5 · R/mR@50 | 6.02 / 4.28 -> 6.08 / 4.38 | Helped | Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation (AAAI 2025 · 2 citations) |
| concatenate RGB and depth frames along the channel dimension | Cityscapes, 2 →28 · FVD | 84 -> 123.8 | Hurt | SyncVP: Joint Diffusion for Synchronous Multi-Modal Video Prediction (CVPR 2025 · 2 citations) |
| Add Propagator, Tracker, and Fuser to the periodically invoked OccInfer baseline | nuScenes, Ninfer = 6 · IoU | 21.10 -> 31.56 | Helped | Exploiting Continuous Motion Clues for Vision-Based Occupancy Prediction (AAAI 2025 · 1 citation) |
| use continuous flow matching instead of discrete masked generation | BDD100K, 60-frame generation · FVD | 533.3 -> 360.9 (12 steps); 571.7 -> 246.0 (30 steps) | Helped | Overcoming Challenges of Long-Horizon Prediction in Driving World Models (NeurIPS 2025 · 1 citation) |
| use SigLIP instead of SSL visual encoders | COIN · step forecasting and task classification accuracy | 47.03 -> 50.05 step forecasting; 90.89 -> 94.38 task classification | Helped | VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning (ICLR 2025 · 1 citation) |
| replace deterministic frame prediction with stochastic frame prediction | DAVIS · J&F_m | 54.4 -> 60.1 | Helped | Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- nuScenes · 3 of 19 papers
- Kinetics-400 · 3 of 19 papers
- Something-Something-V2 · 2 of 19 papers
- MetaWorld · 2 of 19 papers
- TAP-Vid DAVIS · 2 of 19 papers
- TAP-Vid Kubric · 2 of 19 papers
- ScanNet · 2 of 19 papers
- KITTI · 2 of 19 papers
Baselines
- STDiff · 2 of 19 papers
- VC-1 · 2 of 19 papers
- Voltron · 2 of 19 papers
- Cosmos · 2 of 19 papers
- RAFT · 2 of 19 papers
- SEA-RAFT · 2 of 19 papers
- SMURF · 2 of 19 papers
- Doduo · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 49 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 5 papers
- University of Oxford · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Nanyang Technological University · 2 papers
- The University of Hong Kong · 2 papers
- Tongji University · 2 papers
- Peking University · 2 papers
- National University of Singapore · 2 papers
- Nankai University · 2 papers
- Shanghai Jiao Tong University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Layered video decomposition and dynamics (140 papers)
- Physics-inspired motion prediction (113 papers)
- Spatial-temporal video person re-identification (41 papers)
Elsewhere in the topic
- Self-supervised video representation learning (167 papers)
- Video restoration with temporal priors (159 papers)
- Sign language and physiological video analysis (99 papers)
- Weakly supervised video anomaly detection (99 papers)
- Spatio-temporal predictive modeling (91 papers)
- Matching-based video object segmentation (86 papers)
- Learned optical flow estimation (73 papers)
- Learned video compression (68 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).