Research map / 3D vision / Depth and stereo
4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4DTAM, a differentiable-rendering RGB-D SLAM that jointly estimates camera pose and non-rigid surface geometry using 2D Gaussian surfels with an MLP warp field, and introduces the Sim4D synthetic dataset.
Problem
Non-rigid 4D tracking and mapping from an online single RGB-D camera stream, where jointly estimating camera ego-motion and non-rigid scene geometry is high-dimensional and ill-posed.
Approach
Combines 2D Gaussian Splatting surfels (with depth-derived normal initialization and normal regularization) and an MLP warp field with an analytic camera-pose Jacobian, plus ARAP and surface-normal rigidity losses, inside a tracking-and-mapping SLAM framework.
Key results
- On Sim4D, 4DTAM reduces average ATE RMSE to roughly 0.2-1.0 cm across eight sequences, versus 1.45-31.9 cm for SurfelWarp, and improves PSNR on all sequences.
- On Replica static SLAM, the 2DGS variant (MonoGS-2D) reduces average depth L1 from 4.52 cm to 0.54 cm and raises average F1 from 31.9% to 88.8% for the reconstructed mesh.
- The method runs at about 1.5 fps for camera pose estimation, with global optimization finishing in under 1 minute on an RTX 4090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 4DTAM instead of SurfelWarp | Non-rigid RGB-D SLAM · Sim4D | ATE RMSE and L1 depth: ATE RMSE: 6.10 -> 0.25 (curtain); L1 Depth: 49.1 -> 0.96 (curtain) | Helped |
| replace the MonoGS representation with 2DGS (ablation) | Static SLAM · Replica average | Depth L1[cm]↓: 4.52 -> 0.54 | Helped |
| replace the MonoGS representation with 2DGS (ablation) | Static SLAM · Replica average | ATE RMSE[cm]↓: 0.59 -> 0.36 | Helped |
| replace the MonoGS representation with 2DGS (ablation) | Static SLAM · TUM RGB-D average | ATE RMSE[cm]↓: 1.47 -> 1.57 | Hurt |
| replace the MonoGS representation with 2DGS (ablation) | Static SLAM · TUM RGB-D average | Depth L1[cm]↓: 10.7 -> 3.2 | Helped |
| use 2DGS+MLP deformation instead of Morpheus (ablation) | Offline non-rigid reconstruction · KillingFusion | Depth L1[cm]↓: 3.2 -> 4.9 | Hurt |
| use 2DGS+MLP deformation instead of Morpheus (ablation) | Offline non-rigid reconstruction · DeepDeform | Depth L1[cm]↓: 1.9 -> 1.1 | Helped |
| use 2DGS+MLP deformation instead of Morpheus (ablation) | Offline non-rigid reconstruction · iPhone | Depth L1[cm]↓: 2.4 -> 0.57 | Helped |
Limitations
- Tested primarily on small-scale scenes; complex real-world scenes may need 2D priors such as point tracking or optical flow.
- Runs at about 1.5 fps, limiting real-time use.
- Requires depth input (RGB-D); monocular extension only in supplementary material.
Experimental setup
Datasets: Sim4D Replica TUM RGB-D DeepDeform KillingFusion iPhone dataset
Benchmarks: Sim4D Replica TUM RGB-D DeepDeform KillingFusion iPhone dataset
Metrics: ATE RMSE L1 depth error PSNR SSIM LPIPS precision recall F1
Baselines: SurfelWarp MonoGS Morpheus DyNoMo
Builds on: 2D Gaussian Splatting MonoGS 3D Gaussian Splatting DynamicFusion ARAP
Closest papers by meaning
- DG-SLAM: Robust Dynamic Gaussian Splatting SLAM with Hybrid Pose Optimization (NeurIPS 2024 · similarity 0.88)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.88)
- VTGaussian-SLAM: RGBD SLAM for Large Scale Scenes with Splatting View-Tied 3D Gaussians (ICML 2025 · similarity 0.87)
- Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular Stereo and RGB-D Cameras (CVPR 2024 · similarity 0.87)
- Gaussian Splatting SLAM (CVPR 2024 · similarity 0.87)
- CTO-SLAM: Contour Tracking for Object-Level Robust 4D SLAM (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.