Research map / 3D vision / Gaussian splatting
DepthSplat: Connecting Gaussian Splatting and Depth
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DepthSplat, a feed-forward model that fuses multi-view cost volumes with pre-trained Depth Anything V2 monocular features for depth and Gaussian splatting, and shows Gaussian splatting rendering loss can unsupervisedly pre-train depth models.
Problem
Gaussian splatting and depth estimation are studied in isolation; multi-view feature matching for feed-forward 3DGS fails in occlusions, texture-less and reflective regions.
Approach
Augment multi-view cost volumes with pre-trained monocular ViT features (Depth Anything V2) via early concatenation, unproject predicted depth as Gaussian centers, and use the photometric splatting loss as an unsupervised pre-training objective for depth.
Key results
- Achieves 27.47 PSNR / 0.889 SSIM / 0.114 LPIPS on two-view RealEstate10K, beating MVSplat (26.39) and TranSplat (26.69).
- On ScanNet two-view depth, Abs Rel drops to 3.8 with Gaussian splatting pre-training vs 5.9 for UniMatch.
- On DL3DV with 2 input views, PSNR 19.31 vs MVSplat 17.54; cross-dataset DL3DV PSNR 27.66 vs 24.14 for MVSplat.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add monocular depth features to the multi-view cost-volume branch (ablation) | DepthSplat · TartanAir depth | Abs Rel: 12.25 -> 8.46 | Helped |
| Add the multi-view cost-volume branch alongside monocular features (ablation) | DepthSplat · TartanAir depth | Abs Rel: 11.34 -> 8.46 | Helped |
| Add Gaussian splatting pre-training before depth fine-tuning (ablation) | DepthSplat · ScanNet two-view depth | Abs Rel: 4.5 -> 3.8 | Helped |
| Use a larger monocular backbone (ViT-S to ViT-B to ViT-L) (ablation) | DepthSplat, 1-scale · TartanAir depth | Abs Rel: 8.46 -> 6.94 -> 6.07 | Helped |
| Use 2-scale hierarchical matching instead of 1-scale matching (ablation) | DepthSplat with ViT-S · TartanAir depth | Abs Rel: 8.46 -> 7.01 | Helped |
| Use attention fusion instead of concatenation (ablation) | DepthSplat · TartanAir depth | Abs Rel: 8.46 -> 8.40 | No clear effect |
| Pre-train the depth model with Gaussian splatting rendering loss (ablation) | DepthSplat depth model · TartanAir depth | Abs Rel: 10.86 -> 10.20 | Helped |
| Integrate monocular features early with cost volumes | DepthSplat · RealEstate10K two-view novel view synthesis | PSNR: 26.39 -> 27.47 | Helped |
| Use DepthSplat rather than MVSplat for sparse-view reconstruction | DepthSplat · DL3DV 2-view novel view synthesis | PSNR: 17.54 -> 19.31 | Helped |
| Use DepthSplat rather than MVSplat for cross-dataset generalization | RealEstate10K-pretrained models · DL3DV cross-dataset generalization | PSNR: 24.14 -> 27.66 | Helped |
Surprising findings
- Simple channel concatenation of monocular and cost-volume features performs on par with attention-based fusion and better than single-branch ViT cost volumes.
- Gaussian splatting rendering loss alone, without any depth supervision, yields depth pre-training that improves downstream depth error.
Limitations
- Requires known camera poses as input, which is hard to get for extremely sparse views.
- Pixel-aligned Gaussians make the Gaussian count grow large with many input views.
Experimental setup
Datasets: RealEstate10K TartanAir VKITTI2 ScanNet DL3DV KITTI ACID
Benchmarks: ScanNet RealEstate10K DL3DV ACID TartanAir KITTI
Metrics: Abs Rel delta1 RMSE PSNR SSIM LPIPS inference time
Baselines: MVSplat pixelSplat TranSplat MuRF UniMatch DeMoN BA-Net DeepV2D NeuralRecon DRO GPNR AttnRend
Builds on: MVSplat pixelSplat Depth Anything V2 UniMatch 3D Gaussian Splatting
Closest papers by meaning
- MonoSplat: Generalizable 3D Gaussian Splatting from Monocular Depth Foundation Models (CVPR 2025 · similarity 0.92)
- TranSplat: Generalizable 3D Gaussian Splatting from Sparse Multi-View Images with Transformers (AAAI 2025 · similarity 0.92)
- Splatter Image: Ultra-Fast Single-View 3D Reconstruction (CVPR 2024 · similarity 0.90)
- HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction (ICLR 2025 · similarity 0.90)
- FatesGS: Fast and Accurate Sparse-View Surface Reconstruction Using Gaussian Splatting with Depth-Feature Consistency (AAAI 2025 · similarity 0.90)
- SelfSplat: Pose-Free and 3D Prior-Free Generalizable 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.