Research map / Point clouds / Registration and matching
Implicit Correspondence Learning for Image-to-Point Cloud Registration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an implicit correspondence learning method for image-to-point cloud registration that uses a geometric frustum prior for overlap detection, learnable correspondence queries for implicit 2D-3D matching, and an end-to-end pose regression head.
Problem
Image-to-point cloud registration (estimating camera pose of an image within a 3D point cloud) is hindered by unreliable point-wise overlap detection, inaccurate explicit cross-modal feature matching, and an indirect correspondence-based objective that does not optimize pose directly.
Approach
Replace explicit feature-similarity matching and PnP-RANSAC with learnable correspondence queries that yield paired 2D/3D keypoint heatmaps, a frustum-pose-based overlap mask from a geometric prior, and an MLP pose regressor trained end-to-end on pose loss.
Key results
- On KITTI, the method reaches 97.49% Acc with 0.20m RTE and 1.24° RRE, versus 83.04% Acc for VP2P-match.
- On nuScenes, Acc is 90.94% (vs. 88.33% for VP2P-match), with RTE 0.63m and RRE 2.13°.
- Ablations on KITTI show GPDM, ICLM and PRM each contribute; removing ICLM drops Acc from 97.49% to 72.64%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · KITTI | RTE (m): 0.75±1.13 -> 0.20±0.21 | Helped |
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · KITTI | RRE (°): 3.29±7.99 -> 1.24±2.34 | Helped |
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · KITTI | Acc (%): 83.04 -> 97.49 | Helped |
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · nuScenes | RTE (m): 0.89±1.44 -> 0.63±0.44 | Helped |
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · nuScenes | RRE (°): 2.15±7.03 -> 2.13±3.75 | Helped |
| use the full implicit correspondence learning method instead of VP2P-match | Image-to-point cloud registration · nuScenes | Acc (%): 88.33 -> 90.94 | Helped |
| add GPDM (ablation) | Proposed method · KITTI | RTE (m): 0.45±1.05 -> 0.20±0.24 | Helped |
| add GPDM (ablation) | Proposed method · KITTI | Acc (%): 91.89 -> 97.49 | Helped |
| add ICLM and PRM (ablation) | Proposed method · KITTI | Acc (%): 72.64 -> 97.49 | Helped |
| add PRM instead of PnP-RANSAC (ablation) | Proposed method · KITTI | Acc (%): 82.00 -> 97.49 | Helped |
| increase correspondence queries from 16 to 128 (ablation) | ICLM · KITTI | Acc (%): 95.52 -> 97.49 | Helped |
| increase ICLM iteration time from 0 to 3 (ablation) | ICLM · KITTI | Acc (%): 90.01 -> 97.49 | Helped |
Surprising findings
- Removing the ICLM (and PRM) drops Acc sharply to 72.64%, indicating the implicit correspondence module contributes most of the gain.
- Only 16 correspondence queries already surpass VP2P-match on KITTI accuracy (95.52% vs 83.04%).
Limitations
- Evaluated only on KITTI and nuScenes outdoor driving datasets.
- Relies on known camera intrinsics and a frustum-based overlap assumption.
Experimental setup
Datasets: KITTI Odometry nuScenes
Benchmarks: KITTI Odometry nuScenes
Metrics: RTE RRE Acc (RTE<2m and RRE<5°) frustum classification precision/recall/accuracy/F1
Baselines: DeepI2P CorrI2P VP2P-match Grid Cls. + PnP
Builds on: VP2P-match CorrI2P DeepI2P KPFCNN LoFTR
Closest papers by meaning
- FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators (ICLR 2024 · similarity 0.89)
- Partial Point Cloud Registration with Multi-view 2D Image Learning (AAAI 2025 · similarity 0.87)
- 3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration (NeurIPS 2024 · similarity 0.87)
- PointMC: Multi-instance Point Cloud Registration based on Maximal Cliques (ICML 2024 · similarity 0.87)
- Bridge 2D-3D: Uncertainty-aware Hierarchical Registration Network with Domain Alignment (AAAI 2025 · similarity 0.87)
- Zero-shot RGB-D Point Cloud Registration with Pre-trained Large Vision Model (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.