Research map / 3D vision / 3D generation
Learning physics from video
Learns simulators and physical properties from videos, often with Gaussian-based scene representations.
74 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in 3D generation, keyword cluster “motion · interaction · human”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
11 papers in 2023–24 and 54 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.4% to 6.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Newtonian Scene Understanding: Unfolding the Dynamics of Objects in Static Images (CVPR 2016 · 124 citations)
- 2019 · Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces (ICLR 2019 · 16 citations)
- 2021 · SceneGen: Learning To Generate Realistic Traffic Scenes (CVPR 2021 · 96 citations)
- 2022 · DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer (AAAI 2022 · 130 citations)
- 2023 · 3D Assembly Completion (AAAI 2023 · 5 citations)
- 2024 · Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents (CVPR 2024 · 73 citations)
- 2025 · CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images (CVPR 2025 · 13 citations)
- 2026 · PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World (ICML 2026 · 0 citations)
Earliest papers here
- Newtonian Scene Understanding: Unfolding the Dynamics of Objects in Static Images (CVPR 2016 · 124 citations)
- Learning Physical Intuition of Block Towers by Example (ICML 2016 · 75 citations)
- Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces (ICLR 2019 · 16 citations)
What has been tried
48 papers here report 468 measured changes (a component added, removed or swapped, compared with a baseline): helped 73%, no clear effect 5%, hurt 20%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add multi-camera alignment to McNeRF | Waymo Open Dataset novel view synthesis · PSNR | 23.32 -> 25.82 | Helped | Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents (CVPR 2024 · 73 citations) |
| add feature injection | SculptingBench · DINO | 0.848 -> 0.853 | Helped | Image Sculpting: Precise Object Editing with 3D Geometry Control (CVPR 2024 · 21 citations) |
| Add the geometry encoder rather than using concatenated DINO features and an MLP | DeepCAD, single-view · Med CD | 0.056 -> 0.049 | Helped | CADCrafter: Generating Computer-Aided Design Models from Unconstrained Images (CVPR 2025 · 13 citations) |
| use triangle-based structure parameterization instead of min/max bounds | Storage Furniture articulated structure · AID | 36.3 -> 39.1 | Helped | MeshArt: Generating Articulated Meshes with Structure-Guided Transformers (CVPR 2025 · 13 citations) |
| add spatial decomposed modeling | Test set · PSNR | 22.148 -> 25.210 | Helped | MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling (CVPR 2025 · 12 citations) |
| remove the novel-view video synthesizer (add it back) | FluidNexus-Smoke · PSNR (novel view synthesis) | 21.88 -> 30.62 | Helped | FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video (CVPR 2025 · 10 citations) |
| add VEnhancer post-processing | VBench and GPT-4o evaluation · Imaging | 0.666 -> 0.677 | Helped | PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · 10 citations) |
| Add VLM-based data filtering | Articulation-XL · CD-J2J | 2.982 -> 2.586 | Helped | MagicArticulate: Make Your 3D Models Articulation-Ready (CVPR 2025 · 9 citations) |
| add static and dynamic obstacles during training | PointNav · success rate | 68.8% -> 81.6% | Helped | Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation (CVPR 2025 · 9 citations) |
| Replace Transformer blocks with Mamba blocks | CAD sequence reconstruction · A_p | 83.41 -> 99.93 | Helped | Mamba-CAD: State Space Model for 3D Computer-Aided Design Generative Modeling (AAAI 2025 · 6 citations) |
| use optical flow loss instead of render loss | Synthetic dataset (average) · EPE | 1.33 -> 0.51 | Helped | Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation (CVPR 2025 · 5 citations) |
| use a position-conditioned physics code network instead of a learnable per-Gaussian code | Dynamic Object Dataset extrapolation · PSNR | 25.961 -> 31.987 | Helped | FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocity (CVPR 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 35 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Objaverse-XL · 3 of 35 papers
- DeepCAD · 2 of 35 papers
- ModelsResource · 2 of 35 papers
- PartNet · 2 of 35 papers
- PartNet-Mobility · 2 of 35 papers
Baselines
- DeepCAD · 2 of 35 papers
- HNC-CAD · 2 of 35 papers
- PhysGaussian · 2 of 35 papers
- GIC · 2 of 35 papers
- Pinocchio · 2 of 35 papers
- RigNet · 2 of 35 papers
- GVB · 2 of 35 papers
- PAC-NeRF · 2 of 35 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 65 papers; from OpenAlex and the papers' affiliations).
- Nanyang Technological University · 4 papers
- Zhejiang University · 3 papers
- A*STAR · 3 papers
- Stanford University · 3 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Peking University · 2 papers
- Tsinghua University · 2 papers
- Carnegie Mellon University · 2 papers
- Intel · 2 papers
- ByteDance · 2 papers
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Monocular 3D object reconstruction (94 papers)
- Hand-object interaction generation (116 papers)
- Scene-aware human motion synthesis (82 papers)
Elsewhere in the topic
- Learning 3D reconstruction from images (196 papers)
- Diffusion-based novel view synthesis (156 papers)
- Semantic scene completion (153 papers)
- Generative indoor scene synthesis (131 papers)
- Feed-forward 3D asset generation and editing (116 papers)
- Score distillation for text-to-3D (106 papers)
- Open-vocabulary 3D scene understanding (95 papers)
- 2D-to-3D pretrained representations (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).