Research map / Video / Video segmentation and tracking
Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uni4D reconstructs dynamic 4D scenes (camera poses, static/dynamic geometry, dense 3D motion) from a single casual video by energy minimization over outputs of pretrained vision foundation models, with no retraining.
Problem
Recovering camera poses, static and dynamic geometry, and dense 3D motion from a monocular casual video, where no 4D ground-truth data is readily available and joint 4D reasoning is ill-posed.
Approach
A training-free, three-stage divide-and-conquer energy minimization (camera initialization, static bundle adjustment, non-rigid bundle adjustment) that fuses pretrained segmentation, video depth, and dense point-tracking cues with camera-motion and ARAP/smoothness dynamic priors.
Key results
- Camera pose ATE on Sintel of 0.110 (0.092 with known intrinsics), versus 0.108 for MonST3R and 0.137 for CasualSAM.
- On Bonn, pose ATE of 0.017 vs 0.023 for MonST3R; on TUM-dynamics ATE of 0.012 vs 0.108 for MonST3R.
- Depth consistency on Sintel improves SC from 0.109 (UniDepth) to 0.043 with Uni4D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine all optimization stages rather than use stage 1 only (ablation) | Uni4D · Sintel | ATE: 0.150 -> 0.110 | Helped |
| combine all optimization stages rather than use stage 1 only (ablation) | Uni4D · Sintel | RPE trans: 0.051 -> 0.032 | Helped |
| combine all optimization stages rather than use stage 1 only (ablation) | Uni4D · Sintel | RPE rot: 0.551 -> 0.338 | Helped |
| add stage 1 initialization and stage 2 optimization (ablation) | Uni4D · Sintel | ATE: 0.587 -> 0.110 | Helped |
| add stage 1 initialization and stage 2 optimization (ablation) | Uni4D · Sintel | RPE trans: 0.193 -> 0.032 | Helped |
| add stage 1 initialization and stage 2 optimization (ablation) | Uni4D · Sintel | RPE rot: 4.12 -> 0.338 | Helped |
| align depth maps and apply motion priors for depth consistency | Uni4D · Sintel | SC: 0.109 -> 0.043 | Helped |
| align depth maps and apply motion priors for depth consistency | Uni4D · Sintel | δSC<0.01: 31.8 -> 69.3 | Helped |
| align depth maps and apply motion priors for depth consistency | Uni4D · Sintel | δSC<0.05: 76.8 -> 88.1 | Helped |
Surprising findings
- Joint optimization of camera poses in stage 2 alone performs badly (Sintel ATE 0.587); the stage-1 initialization is needed for convergence.
- Uni4D's video depth Abs Rel on Sintel (0.216) is worse than single-frame Metric3D (0.205) and close to UniDepth, so joint methods do not beat single-frame depth models on all depth metrics.
Limitations
- Relies on quality of upstream foundation-model cues; noisy segmentation, depth, or tracks propagate into the result.
- Optimization takes around 5 minutes per 50-frame video, which is slow for long videos.
- Joint depth and pose on Sintel and KITTI depth remain weaker than some single-frame metric depth estimators.
Experimental setup
Datasets: DAVIS Sintel TUM-dynamics Bonn KITTI
Benchmarks: Sintel TUM-dynamics Bonn KITTI DAVIS
Metrics: ATE RPE trans RPE rot Abs Rel delta<1.25 Self-Consistency (SC)
Baselines: DPVO LEAP-VO Robust-CVD CasualSAM MonST3R Metric3D Depth-Pro UniDepth DepthCrafter
Builds on: UniDepthv2 CoTracker3 Grounding-SAM DEVA as-rigid-as-possible
Closest papers by meaning
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.90)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.88)
- Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video (ICLR 2024 · similarity 0.88)
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- Can 4D Foundation Models Remember? (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.