Research map / 3D vision / Depth and stereo
RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free 360° monocular depth estimation that projects ERP images to six cubemap faces, runs a perspective foundation model (Metric3D v2 best), and aligns per-face depth/normals with graph optimization plus per-face scale parameters.
Problem
Monocular depth estimation for 360° images lacks large labeled datasets, and existing optimization-based methods that blend tangent-plane depths from perspective foundation models ignore 3D structural consistency, causing scale inconsistencies across faces.
Approach
Parameterize predicted depth and normal maps from cubemap faces with a graph-based optimization (plane-consistency edges weighted by patch texture similarity, periodic ERP boundary) plus a per-face scale parameter, so depth scale is aligned across faces while preserving 3D structure.
Key results
- On Matterport3D (training-free), RPG360 reaches Chamfer 0.337, F-Score 58.816 and IoU 44.912 versus 0.632 / 28.056 / 16.998 for 360MonoDepth, a 46.7% Chamfer improvement.
- On Stanford2D3D (training-free), RPG360 achieves Chamfer 0.206 and F-Score 73.917 versus 0.576 for 360MonoDepth, a 64.2% Chamfer improvement.
- Finetuning EDM with RPG360 pseudo-labels improves AUC@5° by 5.4% on the 360Loc hall scene and downstream SfM improves by 0.2 to 9.7% AUC@5°.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 360MonoDepth blending with RPG360 graph-based scale alignment | Training-free 360° depth estimation · Matterport3D test | Chamfer: 0.632 -> 0.337 | Helped |
| Replace 360MonoDepth blending with RPG360 graph-based scale alignment | Training-free 360° depth estimation · Stanford2D3D test | Chamfer: 0.576 -> 0.206 | Helped |
| Use Metric3D v2 instead of Omnidata v2 as the foundation model | RPG360 · Matterport3D test | Abs Rel: 0.215 -> 0.203 | Helped |
| Tune EDM with RPG360-generated pseudo ground truth | EDM dense feature matching · 360Loc hall | AUC@5°: 37.83 -> 39.88 | Helped |
| Tune EDM with RPG360-generated pseudo ground truth | EDM dense feature matching · 360Loc atrium, concourse, and piatrium | AUC@5°: 37.19 -> 38.37; 33.13 -> 34.53; 44.45 -> 46.49 | Helped |
| Add graph optimization without a per-face scale parameter (ablation) | RPG360 with Metric3D v2 · Matterport3D reconstruction | Chamfer: 0.380 -> 0.382 | No clear effect |
| Add per-face scale parameter λc to graph optimization (ablation) | RPG360 with Metric3D v2 · Matterport3D reconstruction | Chamfer: 0.382 -> 0.337 | Helped |
| Add per-face scale parameter λc to graph optimization (ablation) | RPG360 with Metric3D v2 · Matterport3D reconstruction | F-Score: 51.605 -> 58.816 | Helped |
| Add per-face scale parameter λc to graph optimization (ablation) | RPG360 with Metric3D v2 · Matterport3D reconstruction | IoU: 37.303 -> 44.912 | Helped |
| Use Marigold as the foundation model rather than Metric3D v2 (ablation) | RPG360 with graph optimization and per-face parameter · Matterport3D reconstruction | Chamfer: 0.337 -> 0.916 | Hurt |
| Use GeoWizard as the foundation model rather than Metric3D v2 (ablation) | RPG360 with graph optimization and per-face parameter · Matterport3D reconstruction | Chamfer: 0.337 -> 0.752 | Hurt |
Surprising findings
- Graph optimization alone with Metric3D v2 gives only marginal 3D gains (Chamfer 0.380 to 0.382); the per-face scale parameter is what yields the major improvement.
- Diffusion-based depth models (Marigold, GeoWizard) produce high-fidelity 2D depth but degrade in 3D point cloud reconstruction, performing worse than Omnidata v2 and Metric3D v2 under the same pipeline.
Limitations
- Less detailed depth for thin structures compared to training-based methods.
- Relies on the quality of the underlying perspective foundation model; 2D metrics lag learning-based methods on Matterport3D.
Experimental setup
Datasets: Matterport3D Stanford2D3D 360Loc Structured3D
Benchmarks: Matterport3D Stanford2D3D 360Loc Matterport3D-2K
Metrics: Chamfer distance F-Score IoU Abs Rel RMSE delta threshold accuracy AUC@5°
Baselines: 360MonoDepth Depth Anywhere Elite360D UniFuse ACDNet BiFuse++ SliceNet Peng and Zhang
Builds on: 360MonoDepth Depth Anywhere Metric3D v2 EDM IM360
Closest papers by meaning
- Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation (NeurIPS 2024 · similarity 0.90)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.88)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.86)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.86)
- HELVIPAD: A Real-World Dataset for Omnidirectional Stereo Depth Estimation (CVPR 2025 · similarity 0.86)
- DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.