Research map / 3D vision / Depth and stereo
Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a perspective depth foundation model (Depth Anything) into 360-degree monocular depth estimators via six-face cube projection pseudo-labels on unlabeled panoramas, with random rotation and SAM-based sky/watermark masking, trained jointly with labeled data.
Problem
360-degree monocular depth estimation is limited by scarce labeled equirectangular depth data, and perspective depth models transfer poorly to equirectangular images due to projection distortion and domain gap.
Approach
Generate pseudo depth labels for unlabeled 360 images by projecting to six cube faces, running a frozen perspective teacher (Depth Anything) on the faces, and jointly training a 360 model on half labeled/half pseudo-labeled batches; random rotation of the equirectangular image before cube projection removes cube-face artifacts, and SAM/Grounding-DINO masks remove sky and watermark regions.
Key results
- Zero-shot on Stanford2D3D, BiFuse++ trained on Matterport3D with Structured3D pseudo-labels reaches AbsRel 0.082 and delta1 0.931, versus 0.090 and 0.921 without pseudo-labels.
- On the Matterport3D in-domain test, BiFuse++ with Structured3D pseudo-labels reaches AbsRel 0.085 and delta1 0.917 versus 0.094 and 0.914 baseline.
- Depth Anything applied directly to Stanford2D3D zero-shot yields AbsRel 0.248, far worse than the 360 models trained with the proposed pipeline (about 0.082-0.090).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pseudo-labeled Structured3D training data | UniFuse · Matterport3D | Abs Rel: 0.102 -> \underline{0.089} | Helped |
| Add pseudo-labeled Structured3D training data | BiFuse++ · Matterport3D | Abs Rel: 0.094 -> \textbf{0.085} | Helped |
| Add pseudo-labeled Structured3D training data | UniFuse · Stanford2D3D test | Abs Rel: 0.090 -> \underline{0.086} | Helped |
| Add pseudo-labeled SpatialAudioGen training data | UniFuse · Stanford2D3D test | Abs Rel: 0.090 -> 0.090 | No clear effect |
| Add pseudo-labeled Structured3D training data | BiFuse++ · Stanford2D3D test | Abs Rel: 0.090 -> \textbf{0.082} | Helped |
| Add pseudo-labeled SpatialAudioGen training data | BiFuse++ · Stanford2D3D test | Abs Rel: 0.090 -> \underline{0.086} | Helped |
| Add pseudo-labeled Structured3D training data | UniFuse · Structured3D test | Abs Rel: 0.202 -> 0.130 | Helped |
| Add pseudo-labeled SpatialAudioGen training data | UniFuse · Structured3D test | Abs Rel: 0.202 -> 0.152 | Helped |
| Initialize with the proposed relative-depth model before one-epoch metric-depth fine-tuning | UniFuse · Stanford2D3D metric depth fine-tuning | Abs Rel: 0.111 -> 0.118 | Hurt |
| Change GT:pseudo ratio from 1:1 to 1:2 (ablation) | UniFuse · Stanford2D3D test | Abs Rel: 0.086 -> 0.087 | Hurt |
| Change GT:pseudo ratio from 1:1 to 1:4 (ablation) | UniFuse · Stanford2D3D test | Abs Rel: 0.086 -> 0.085 | No clear effect |
| Replace cube projection with tangent projection (ablation) | UniFuse · Stanford2D3D test | Abs Rel: \textbf{0.086} -> 0.087 | No clear effect |
Surprising findings
- Directly applying Depth Anything pseudo-labels on cube-projected data without random rotation produces cube artifacts and does not improve performance.
- Pseudo-labels from SpatialAudioGen (unlabeled) give similar gains to those from labeled-dataset Structured3D pseudo-labels.
Limitations
- Heavy reliance on quality of unlabeled data and perspective teacher pseudo-labels; training produced NaN values without data cleaning.
- Scale ambiguity: aligning perspective patches back to equirectangular with consistent scale is unsolved, so loss is computed in perspective domain.
- Scarcity of 360 data remains compared with other tasks even with unlabeled data.
Experimental setup
Datasets: Matterport3D Stanford2D3D Structured3D SpatialAudioGen
Benchmarks: Matterport3D Stanford2D3D Structured3D
Metrics: AbsRel delta1 delta2 delta3 MAE RMSE RMSElog
Baselines: UniFuse BiFuse++ BiFuse SliceNet HRDFuse HoHoNet EGFormer Depth Anything Marigold
Builds on: Depth Anything UniFuse BiFuse++ Segment Anything MiDaS
Closest papers by meaning
- RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization (NeurIPS 2025 · similarity 0.90)
- Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data (CVPR 2024 · similarity 0.89)
- Depth Anything V2 (NeurIPS 2024 · similarity 0.88)
- DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images (NeurIPS 2026 · similarity 0.88)
- Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth (NeurIPS 2026 · similarity 0.88)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.