Research map / 3D vision / Depth and stereo
UniK3D: Universal Camera Monocular 3D Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniK3D, a generalizable monocular metric 3D estimator that models any camera (pinhole, fisheye, panoramic) via a spherical 3D output and a learned spherical-harmonics camera ray representation.
Problem
Monocular metric 3D estimation typically assumes pinhole or rectified cameras, which fails for fisheye and panoramic images with large field of view and strong distortion.
Approach
Predicts a fully spherical 3D output (angles plus radial distance) with camera rays represented as a learned superposition of spherical harmonics, plus an asymmetric quantile angular loss and camera-conditioning strategies (static encoding, curriculum, gradient detach) to prevent FoV contraction.
Key results
- Zero-shot L.FoV domain: UniK3D-Large reaches delta1 SSI 91.2% and FA 71.6%, outperforming the second-best method by over 20% and 40% respectively.
- Stanford-2D3D panoramic zero-shot: UniK3D reaches delta1 96.8 and AbsRel 8.01, beating UniFuse (91.3 / 9.42).
- S.FoV zero-shot: UniK3D-Large achieves delta1 SSI 96.1 vs. UniDepth 94.9, while staying top-ranked on conventional pinhole data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the pinhole camera representation with SH camera rays (ablation) | UniK3D · L.FoV and Pano | FA: 45.2 -> 53.5; 24.6 -> 58.6 | Helped |
| use non-parametric rays instead of SH camera rays (ablation) | UniK3D · Pano | FA: 58.6 -> 51.7 | Hurt |
| use SH camera rays instead of Zernike-polynomial coefficients (ablation) | UniK3D · Pano | FA: 31.8 -> 58.6 | Helped |
| predict spherical radius instead of Cartesian depth with SH camera rays (ablation) | UniK3D · Pano | FA: 10.9 -> 53.8 | Helped |
| predict spherical radius instead of Cartesian depth with SH camera rays (ablation) | UniK3D · S.FoVDist | FA: 42.1 -> 35.0 | Hurt |
| add strongly distorted-camera training images (ablation) | UniK3D with pinhole data and output · S.FoVDist | FA: 31.7 -> 40.4 | Helped |
| add strongly distorted-camera training images (ablation) | UniK3D with SH camera and depth output · Pano | ρA: 16.1 -> 14.8 | Hurt |
| add asymmetric angular loss (ablation) | UniK3D with radius output and SH camera model · Pano | FA: 53.8 -> 56.1 | Helped |
| add enhanced camera conditioning (ablation) | UniK3D with asymmetric angular loss · S.FoVDist | FA: 39.5 -> 44.6 | Helped |
| add enhanced camera conditioning (ablation) | UniK3D with asymmetric angular loss · Pano | FA: 56.1 -> 58.6 | Helped |
| use UniK3D-Large instead of the strongest listed baseline | zero-shot monocular 3D estimation · L.FoV | FA: 29.7 -> 71.6 | Helped |
| use UniK3D-Large instead of the strongest listed baseline | zero-shot camera estimation · Pano | ρA: 2.1 -> 57.1 | Helped |
Surprising findings
- Non-parametric per-pixel ray prediction underperforms the SH camera model on wide-FoV and panoramic settings, suggesting it needs much more data for tail distributions.
- Radius-based output helps wide-FoV and panoramic reconstruction but degrades small-FoV distorted views relative to depth output.
Limitations
- Panoramic performance limited by the difficulty of representing panoramic images with log-depth representation and by limited training data diversity for wide FoV.
- Authors note that expanding the diversity and coverage of training data could further improve robustness.
- Radius output is more sensitive to small angular variations, hurting small-FoV distorted views.
Experimental setup
Datasets: A2D2 aiMotive Argoverse2 ARKit-Scenes ASE BEDLAM BlendedMVS DL3DV DrivingStereo DynamicReplica EDEN FutureHouse
Benchmarks: NYU-Depth V2 KITTI nuScenes IBims-1 ETH3D Diode Indoor ADT ScanNet++ (DSLR) KITTI360 Stanford-2D3D
Metrics: delta1 SSI FA (F1 AUC) rho_A (camera angular AUC) delta1 AbsRel
Baselines: DepthAnything DepthAnythingv2 Metric3D Metric3Dv2 ZoeDepth UniDepth MASt3R DepthPro BiFuse BiFuse++ UniFuse
Builds on: UniDepth DINOv2 quantile regression
Closest papers by meaning
- UniDepth: Universal Monocular Metric Depth Estimation (CVPR 2024 · similarity 0.91)
- UniSHARP: Universal Sharp Monocular View Synthesis (NeurIPS 2026 · similarity 0.88)
- DUSt3R: Geometric 3D Vision Made Easy (CVPR 2024 · similarity 0.87)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.87)
- A Simple yet Universal Framework for Depth Completion (NeurIPS 2024 · similarity 0.87)
- DepthMaster: Unified Monocular Depth Estimation for Perspective and Panoramic Images (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.