Research map / Video / Video segmentation and tracking
DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DynamicGen, an automated pipeline that turns internet monocular videos into metric-scale 4D annotations (point maps, camera poses, intrinsics, masklets, and hierarchical captions), and releases the DynamicVerse dataset of 100K+ videos.
Problem
Real-world 4D datasets with metric-scale geometry, dynamic object masks, and descriptive captions are scarce; existing ones come from limited simulators or up-to-scale SfM and lack rich captions, limiting foundation models on monocular internet videos.
Approach
A multi-stage dynamic bundle adjustment (dynamic masking, coarse camera init, static and non-rigid BA, flow-based sliding-window refinement) that combines foundation models (UniDepthV2, CoTracker3, UniMatch, Qwen2.5-VL, SA2VA, DAM) to produce metric-scale 4D annotations plus three-level captions (object, scene, camera) from raw videos.
Key results
- Video depth on Sintel: DynamicGen reaches Abs 0.205 and delta1.25 72.9, best among compared methods (Uni4D 0.216/72.5 in the same category, Metric3D 0.205/71.9 single-frame).
- Camera pose on Sintel: DynamicGen achieves ATE 0.108, RPE trans 0.029, RPE rot 0.282, the best of the joint depth-and-pose methods listed.
- Camera intrinsics on Sintel: AFE 413.1 px and RFE 0.241%, versus UniDepth 447.4 / 0.357 and Dust3r 434.0 / 0.364.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynamicGen instead of Uni4D | video depth estimation · Sintel | Abs: 0.216 -> 0.205 | Helped |
| Use DynamicGen instead of Uni4D | video depth estimation · KITTI | δ1.25: 89.7 -> 91.2 | Helped |
| Use DynamicGen instead of Uni4D | camera pose estimation · Sintel | RPE rot: 0.338 -> 0.282 | Helped |
| Use DynamicGen instead of Uni4D | camera pose estimation · TUM-dynamics | ATE: 0.012 -> 0.012 | No clear effect |
| Use DynamicGen instead of Dust3r | camera intrinsics estimation · Sintel | RFE (%): 0.364 -> 0.241 | Helped |
| Add semantic-aware key frame extraction (ablation) | dynamic scene caption generation · SA-V caption sample | Avg.: 77.37 -> 78.29 | Helped |
| Add hierarchical prompting (ablation) | dynamic scene caption generation · SA-V caption sample | Avg.: 78.29 -> 79.43 | Helped |
| Add caption rephrasing (ablation) | dynamic scene caption generation · SA-V caption sample | Avg.: 79.43 -> 79.53 | Helped |
| Add Chain-of-Thought prompting (ablation) | dynamic scene caption generation · SA-V caption sample | Avg.: 79.53 -> 81.97 | Helped |
| Add epipolar-mask dynamics filtering (ablation) | dynamic bundle adjustment · Sintel | ATE: 0.114694 -> 0.114065 | Helped |
| Add VLM-based semantic dynamics analysis (ablation) | dynamic bundle adjustment · Sintel | ATE: 0.114694 -> 0.11053 | Helped |
| Add optical flow-based sliding-window global refinement (ablation) | dynamic bundle adjustment · Sintel | Abs: 0.216433 -> 0.214282 | Helped |
Surprising findings
- Video depth on KITTI: Metric3D (Abs 0.039) and Depth-pro outperform DynamicGen (Abs 0.091), contradicting the claim of best performance across all datasets and metrics.
Limitations
- Noise and quality variance from in-the-wild internet videos can compromise metric-scale geometry and motion recovery in cluttered or occluded scenes.
- High computational overhead of large-scale models on long videos limits real-time performance and scalable deployment.
- Generalization is bounded by the biases and capabilities of the underlying foundation models.
Experimental setup
Datasets: DAVIS2017 Youtube-VIS UVO-dense VOST BURST MOSE SA-V PointOdyssey Spring Dynamic Replica MVS-Synth RealCam-Vid
Benchmarks: Sintel KITTI TUM-dynamics HyperNeRF
Metrics: Abs Rel delta1.25 ATE RPE trans RPE rot AFE RFE Accuracy vIoU G-VEval ACCR scores
Baselines: Monst3r Uni4D Depth-pro Metric3D DepthCrafter Robust-CVD CasualSAM DPVO LEAP-VO UniDepth Dust3r 4D-LangSplat
Builds on: Uni4D UniDepthV2 CoTracker3 UniMatch SA2VA
Closest papers by meaning
- 4DWorldBench: A Comprehensive Evaluation Framework for 3D/4D World Generation Models (CVPR 2026 · similarity 0.89)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.87)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.87)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.87)
- RoboScape: Physics-informed Embodied World Model (NeurIPS 2025 · similarity 0.86)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.