Research map / Point clouds / Point cloud classification
Flexible Depth Completion for Sparse and Varying Point Densities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Affinity-Based Shift Correction (ASC), a decoder-stage module that iteratively aligns depth predictions to sparse input depth using learned pixel-point affinities, enabling robust depth completion across sparse and variable point densities and joint estimation/completion.
Problem
Depth completion methods tuned for dense inputs (64-line LiDAR, 500 NYUv2 points) degrade on very sparse (e.g., 4-line LiDAR, 32 points) and variable or irregular depth distributions, and transfer poorly across datasets.
Approach
A transformer-derived affinity between every image pixel and every input depth point lets each depth point adaptively influence a large image region; predicted depth errors at input points are propagated via these affinities to shift-correct the depth map at each decoder stage, with a learned confidence map choosing between corrected and initial predictions. The module plugs into any encoder-decod
Key results
- On KITTI variable scan lines, Ours (NLSPN Base) reaches 2051.9 mm RMSE at 1 line vs NLSPN 2195.5 mm, with gains largest in fewer-line settings.
- On NYUv2 with 2 random points (variable training), REL drops to 0.1272 (NLSPN Base) and 0.0977 with Effb5 backbone, vs NLSPN 0.1321.
- Ours (R34 RGB) on nuScenes 8 lines fixed training yields RMSE 10877 mm vs NLSPN 14099 mm; adding ASC to BTS/AdaBins enables joint estimation and completion with one model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Affinity-Based Shift Correction to NLSPN | NLSPN · KITTI 4 Line, Fixed Scan Line | RMSE: 1851.2 -> 1798.1 | Helped |
| add Affinity-Based Shift Correction to NLSPN | NLSPN · NYUv2 2 Points, Variable # of Points | REL: 0.1321 -> 0.1272 | Helped |
| add Affinity-Based Shift Correction to NLSPN | NLSPN · NYUv2 800 SIFT Keypoints | REL: 0.036 -> 0.028 | Helped |
| add Affinity-Based Shift Correction to NLSPN | NLSPN · VOID 150 Points | RMSE: 217.0 -> 181.5 | Helped |
| use an affinity-weighted sum of input depths instead of depth point features (ablation) | ASC module · NYUv2 2 Points | REL: 0.106 -> 0.136 | Hurt |
| add affinity-weighted depth point features to the affinity-weighted depth errors (ablation) | ASC module · NYUv2 2 Points | REL: 0.130 -> 0.115 | Hurt |
| add distance-to-nearest-point input to correction confidence prediction (ablation) | ASC module · NYUv2 2 Points | REL: 0.113 -> 0.108 | Helped |
| add initial feature-only fusion in the first correction module (ablation) | ASC module · NYUv2 2 Points | REL: 0.108 -> 0.102 | Helped |
| use ℓ1 loss for supervising Dfinal and Dfuse (ablation) | ASC module · NYUv2 2 Points | REL: 0.102 -> 0.098 | Helped |
| use RGB rather than RGBD encoder input (ablation) | ResNet34 depth completion model · NYUv2 2 Points | REL: 0.134 -> 0.110 | Helped |
| add the NLSPN refinement head (ablation) | ResNet34+ RGBD with ASC · NYUv2 500 Points | REL: 0.0140 -> 0.0130 | Helped |
| train one model on variable rather than fixed scan lines | Ours (NLSPN Base) · KITTI 4 Line | RMSE: 1798.1 -> 1829.0 | Mixed |
Surprising findings
- RGB-only encoder input with the correction module outperforms RGBD-input completion models for sparse inputs, while RGBD is better for dense inputs.
- Fusing affinity-weighted features and depth errors makes the model over-confident in corrected depth for very few points, requiring correction confidence prediction.
Limitations
- Shift correction on the coarsest decoder stage is unreliable due to scale ambiguity, requiring feature-only fusion there.
- With fewer than five input points, corrected predictions can be worse than initial predictions in some regions, requiring confidence weighting.
- Trade-off: RGB-input encoders excel for sparse input but RGBD+NLSPN is better for dense (500-point) inputs.
Experimental setup
Datasets: KITTI NYUv2 VOID nuScenes SIFT keypoints
Benchmarks: KITTI depth completion NYUv2 depth completion VOID nuScenes transfer SIFT keypoints NYUv2
Metrics: RMSE MAE REL delta1.25 delta1.02 delta1.05 delta1.10
Baselines: MultiStack GuideNet NLSPN ENet SparseFormer CostDCNet SparseSPN SpAgNet PointFusion S2D CSPN
Builds on: NLSPN BTS AdaBins SparseFormer CSPN
Closest papers by meaning
- Distilling Monocular Foundation Model for Fine-grained Depth Completion (CVPR 2025 · similarity 0.89)
- Zero-shot Depth Completion via Test-time Alignment with Affine-invariant Depth Prior (AAAI 2025 · similarity 0.89)
- A Simple yet Universal Framework for Depth Completion (NeurIPS 2024 · similarity 0.89)
- Masked Spatial Propagation Network for Sparsity-Adaptive Depth Refinement (CVPR 2024 · similarity 0.88)
- Mind The Edge: Refining Depth Edges in Sparsely-Supervised Monocular Depth Estimation (CVPR 2024 · similarity 0.88)
- Beyond Virtual Points: Depth-Enhanced LiDAR-only 3D Object Detection with Semi-Supervised Learning (Student Abstract) (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.