Research map / Label-efficient and robust learning / Test-time adaptation
Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a zero-shot depth completion method that aligns an affine-invariant depth diffusion prior (Marigold/DepthFM) with sparse metric depth at test time via optimization loops inside guided diffusion sampling, plus prior-based outlier filtering and an R-SSIM loss.
Problem
Depth completion from sparse measurements is ill-posed and learned models generalize poorly to out-of-domain scenes; the goal is domain-generalizable, training-free depth completion.
Approach
Use a pre-trained affine-invariant monocular depth diffusion model as a prior and enforce sparse metric measurements as hard constraints via a test-time optimization loop on the predicted clean latent during reverse sampling, with superpixel/RANSAC-based outlier filtering and a relative SSIM structure loss.
Key results
- On domain generalization, Ours (+Marigold) achieves RMSE 0.149 on NYUv2 and 1.561 on nuScenes, versus 1.755 for the best prior TTA method ProxyTTA.
- Ours (+Marigold) reaches RMSE 1.413 on KITTI DC without training on it, comparable to unsupervised in-domain methods (e.g., KBNet 1.126 on KITTI DC), and 0.111 on VOID vs KBNet 0.095.
- Ablation and efficiency: DepthFM with 1 step runs in 16s versus 101s for Marigold with 50 steps on KITTI validation, at RMSE 1.601 vs 1.413.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use test-time alignment with an affine-invariant depth prior | Zero-shot depth completion · NYUv2, SceneNet, Waymo, nuScenes | RMSE and MAE | Helped |
| Use DepthFM instead of Marigold as the depth prior | Zero-shot depth completion · NYUv2 and SceneNet | RMSE: 0.149 -> 0.145; 0.207 -> 0.178 | Mixed |
| Use Marigold instead of DepthFM as the depth prior | Zero-shot depth completion · Waymo and nuScenes | RMSE: 2.162 -> 2.115; 1.622 -> 1.561 | Helped |
| Use Marigold instead of DepthFM as the depth prior | Zero-shot depth completion · Waymo | MAE: 1.133 -> 1.121 | Hurt |
| Use Marigold instead of DepthFM as the depth prior | Zero-shot depth completion · NYUv2 | MAE: 0.077 -> 0.059 | Helped |
| Use prior-based outlier filtering instead of Window Filter | Sparse depth filtering · KITTI DC validation set, sparse | AUC: 0.3480 -> 0.2959 | Helped |
| Use prior-based outlier filtering instead of Window Filter | Semi-dense depth filtering · KITTI DC validation set, semi-dense | AUC: 0.9629 -> 0.5103 | Helped |
| Use prior-based filtering instead of manual filtering (ablation) | Zero-shot depth completion · KITTI DC | RMSE: 1.198 -> 1.413 | Hurt |
| Use prior-based filtering instead of manual filtering (ablation) | Zero-shot depth completion · KITTI DC | MAE: 0.287 -> 0.397 | Hurt |
| Use prior-based filtering instead of manual filtering (ablation) | Zero-shot depth completion · VOID | RMSE: 0.112 -> 0.111 | Helped |
| Use prior-based filtering instead of manual filtering (ablation) | Zero-shot depth completion · VOID | MAE: 0.045 -> 0.044 | Helped |
| Add test-time alignment correction steps rather than using guided sampling (ablation) | Zero-shot depth completion · VOID and KITTI DC | RMSE and MAE | Helped |
Surprising findings
- Guided sampling with sparse measurements often corrupts the completed depth map due to diffusion stochasticity, motivating the hard-constraint alignment step.
- Ground truth for nuScenes generated by 3D-lifting with off-the-shelf segmentation can be physically inaccurate, inflating RMSE values of 5-6 m for some methods.
Limitations
- Standard guided latent diffusion sampling makes inference slow (Marigold 50 steps ~101s per sample).
- Performance depends on the quality of sparse measurements; outlier filtering benefits outdoor more than indoor and manual filtering is needed for some cases.
- Ground truth quality in nuScenes is questionable, making evaluation sensitive to GT processing.
Experimental setup
Datasets: NYUv2 SceneNet Waymo nuScenes KITTI DC VOID
Benchmarks: NYUv2 SceneNet Waymo nuScenes KITTI DC VOID
Metrics: RMSE MAE AUC
Baselines: BNAdapt CoTTA ProxyTTA VPP4DC DepthPrompting UniDC Self-S2D VOICED ScaffNet KBNet SPTR
Builds on: Marigold DepthFM guided diffusion sampling ProxyTTA
Closest papers by meaning
- A Simple yet Universal Framework for Depth Completion (NeurIPS 2024 · similarity 0.90)
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (ICLR 2025 · similarity 0.90)
- Flexible Depth Completion for Sparse and Varying Point Densities (CVPR 2024 · similarity 0.89)
- DepthFM: Fast Generative Monocular Depth Estimation with Flow Matching (AAAI 2025 · similarity 0.88)
- Test-Time Adaptation for Depth Completion (CVPR 2024 · similarity 0.88)
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.