Research map / Video / Video segmentation and tracking
Physics-inspired motion prediction
Learns interaction dynamics and motion simulators from video for future motion forecasting.
113 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Video segmentation and tracking, keyword cluster “prediction · person · pose”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 36 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 5.0% to 6.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Temporal Epipolar Regions (CVPR 2016 · 1 citation)
- 2017 · Visual Interaction Networks: Learning a Physics Simulator from Video (NeurIPS 2017 · 179 citations)
- 2018 · Convolutional Sequence to Sequence Model for Human Dynamics (CVPR 2018 · 360 citations)
- 2019 · Learning 3D Human Dynamics From Video (CVPR 2019 · 563 citations)
- 2020 · Understanding Human Hands in Contact at Internet Scale (CVPR 2020 · 253 citations)
- 2021 · Multimodal Motion Prediction With Stacked Transformers (CVPR 2021 · 475 citations)
- 2022 · Spatio-Temporal Gating-Adjacency GCN for Human Motion Prediction (CVPR 2022 · 130 citations)
- 2023 · TrEP: Transformer-Based Evidential Prediction for Pedestrian Intention with Uncertainty (AAAI 2023 · 68 citations)
- 2024 · Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · 104 citations)
- 2025 · Bridging Traffic State and Trajectory for Dynamic Road Network and Trajectory Representation Learning (AAAI 2025 · 17 citations)
- 2026 · WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling (ICML 2026 · 0 citations)
Earliest papers here
- Temporal Epipolar Regions (CVPR 2016 · 1 citation)
- Unsupervised Learning for Physical Interaction through Video Prediction (NeurIPS 2016)
- Visual Interaction Networks: Learning a Physics Simulator from Video (NeurIPS 2017 · 179 citations)
What has been tried
93 papers here report 913 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 4%, hurt 19%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add layout condition | nuScenes val multiview video generation · FID | 18.9 -> 15.8 | Helped | Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · 104 citations) |
| Add PSM | Xia dataset · CC | 37.4 -> 19.7 | Helped | MoST: Motion Style Transformer Between Diverse Action Contents (CVPR 2024 · 21 citations) |
| Add traffic-state dynamics modeling | Xi'an MSTSP and TTE · performance | Helped | Bridging Traffic State and Trajectory for Dynamic Road Network and Trajectory Representation Learning (AAAI 2025 · 17 citations) | |
| replace separate modeling with Causal Interactive Injection | InterHuman · R-Precision Top 1 | 0.371±.010 -> 0.491±.005 | Helped | TIMotion: Temporal and Interactive Framework for Efficient Human-Human Motion Generation (CVPR 2025 · 10 citations) |
| Add Cross-Attention Prototyping to Base | Sintel flow ablation · Clean EPE | 0.55 -> 0.51 | Helped | Prototypical Transformer As Unified Motion Learners (ICML 2024 · 8 citations) |
| remove decoder cross-attention to context frames for future-frame reconstruction | context-dynamics decoupling visualization · context reconstruction | Hurt | iVideoGPT: Interactive VideoGPTs are Scalable World Models (NeurIPS 2024 · 8 citations) | |
| Use progressive sampling rather than small-range sampling | Minecraft · PSNR | 19.23 -> 23.98 | Helped | WorldMem: Long-term Consistent World Simulation with Memory (NeurIPS 2025 · 7 citations) |
| Add Symplectic Temporal Aggregation (STA) | Human3.6M Eating, 1,000 ms · MPJPE | 65.1 -> 52.2 | Helped | Rethinking Human Motion Prediction with Symplectic Integral (CVPR 2024 · 5 citations) |
| Add the proposed clustering approach to DMoN | 3ET+ · p10 | +14% | Helped | EyeGraph: Modularity-aware Spatio Temporal Graph Clustering for Continuous Event-based Eye Tracking (NeurIPS 2024 · 5 citations) |
| Add batch slot-slot contrast to feature reconstruction | MOVi-C · FG-ARI | 49.7 -> 69.3 | Helped | Temporally Consistent Object-Centric Learning by Contrasting Slots (CVPR 2025 · 4 citations) |
| Add HVM and HIM-TRN | H3.6M · MPJPE | 16.8 -> 7.8; 121.9 -> 94.1 | Helped | HVIS: A Human-like Vision and Inference System for Human Motion Prediction (AAAI 2025 · 3 citations) |
| add multi-source input rather than use only radar | SEVIR, 12 →36 frames · mCSI | 0.2377 -> 0.2562 | Helped | Integrating Multi-Source Data for Long Sequence Precipitation Forecasting (AAAI 2025 · 3 citations) |
What reviewers will expect
Datasets and baselines named in the 25 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- DAVIS-2017 · 2 of 25 papers
Baselines
- STEVE · 2 of 25 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 58 papers; from OpenAlex and the papers' affiliations).
- Zhejiang University · 5 papers
- Meta · 5 papers
- Peking University · 4 papers
- A*STAR · 2 papers
- Purdue University · 2 papers
- Shanghai AI Laboratory · 2 papers
- Tianjin University · 2 papers
- Stanford University · 2 papers
- University of Texas at Austin · 2 papers
- Shanghai Jiao Tong University · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Layered video decomposition and dynamics (140 papers)
- Stochastic video prediction (125 papers)
- Spatial-temporal video person re-identification (41 papers)
Elsewhere in the topic
- Self-supervised video representation learning (167 papers)
- Video restoration with temporal priors (159 papers)
- Sign language and physiological video analysis (99 papers)
- Weakly supervised video anomaly detection (99 papers)
- Spatio-temporal predictive modeling (91 papers)
- Matching-based video object segmentation (86 papers)
- Learned optical flow estimation (73 papers)
- Learned video compression (68 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).