Research map / 3D vision / 3D generation
Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Motion4D lifts 2D foundation-model priors (SAM2 masks, point tracks, depth) into a dynamic 4D Gaussian Splatting representation with iterative motion and semantic refinement, improving 3D-consistent segmentation, point tracking, and novel view synthesis on monocular videos.
Problem
2D foundation models (SAM2, TAP-style trackers, depth models) produce spatially misaligned and temporally flickering predictions on dynamic monocular videos because they lack 3D consistency.
Approach
Couples a dynamic 3D Gaussian Splatting model with motion and semantic fields, using a learned 3D uncertainty field to weight 2D track/depth priors, error-driven adaptive resampling, and iteratively feeding rendered 3D masks back as prompts to SAM2, organized into sequential chunk-wise and global optimization stages.
Key results
- On DyCheck-VOS, Motion4D + SAM2 reaches 91.7 J&F vs 89.4 for SAM2 alone.
- On DyCheck 2D point tracking, Motion4D gets 37.3 AJ vs 31.0 for CoTracker3 and 34.4 for Shape of Motion.
- On DyCheck novel view synthesis, Motion4D reports 17.91 PSNR, 0.69 SSIM, 0.42 LPIPS, improving over Shape of Motion (16.72 PSNR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add iterative updates to semantic masks and confidence-based point-track refinement (ablation) | Motion4D · DyCheck-VOS | J&F: 87.6 -> 91.7 | Helped |
| add error-driven adaptive sampling (ablation) | Motion4D · DyCheck-VOS | J&F: 88.9 -> 91.7 | Helped |
| use sequential temporal-chunk initialization instead of full-video initialization (ablation) | Motion4D · DyCheck-VOS | J&F: 88.0 -> 91.7 | Helped |
| add global joint optimization over the full sequence (ablation) | Motion4D · DyCheck-VOS | J&F: 90.3 -> 91.7 | Helped |
| use Motion4D instead of Shape of Motion | DyCheck 2D Tracking · DyCheck | AJ: 34.4 -> 37.3 | Helped |
| use Motion4D instead of Shape of Motion | DyCheck 3D Point Tracking · DyCheck | EPE: 0.082 -> 0.072 | Helped |
| use Motion4D instead of SAM2 | video object segmentation · DAVIS 2017 val | J&F: 90.7 -> 90.8 | No clear effect |
Surprising findings
- Rendered Motion4D masks score slightly below SAM2 on DAVIS (89.7 vs 90.7 J&F), attributed to reconstruction quality limiting boundary detail.
- Prior 3D-lifting baselines Semantic Flow and SADG score below 2D SAM2 on DyCheck-VOS when given SAM2 masks, suggesting inconsistent 2D inputs degrade 3D methods.
Limitations
- Reconstruction quality degrades under severe occlusions, low-texture regions, or inaccurate depth, which propagates to motion and semantic predictions.
- Relies heavily on quality of underlying 3D reconstruction and 2D priors.
Experimental setup
Datasets: DyCheck DAVIS 2017
Benchmarks: DyCheck-VOS DAVIS 2017 val DAVIS 2D tracking DyCheck 2D tracking DyCheck 3D tracking DyCheck novel view synthesis
Metrics: J&F J F AJ <delta_avg OA EPE delta_3D 0.05 delta_3D 0.10 PSNR
Baselines: XMem DEVA SAM2 Semantic Flow SADG TAPIR LocoTrack CoTracker BootsTAPIR CoTracker3 HyperNeRF Deformable-3D-GS
Builds on: 3D Gaussian Splatting SAM2 Shape of Motion Semantic Flow
Closest papers by meaning
- 4DP-QA: Scalable QA for 4D Perception in Vision Language Models (CVPR 2026 · similarity 0.91)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.90)
- Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video (CVPR 2025 · similarity 0.90)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.90)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.