Research map / Point clouds / Registration and matching
FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeReg, a training-free image-to-point cloud registration method that unifies modalities via depth-to-image ControlNet/Stable Diffusion features and Zoe-Depth-based FCGF geometric features, fused for NN matching and Kabsch/PnP pose estimation.
Problem
Cross-modality feature matching between images and point clouds for image-to-point cloud registration is hard to learn with metric learning, which yields poor robustness and generalization.
Approach
Instead of metric learning across modalities, map point clouds into the image modality via depth-to-image diffusion (ControlNet) intermediate features, which are semantically consistent with RGB diffusion features, and fuse them with geometric features computed on monocular metric depth from Zoe-Depth, requiring no I2P training.
Key results
- On 3DMatch, FreeReg achieves IR 47.0% vs 24.9% for I2P-Matr and IN 82.8 vs 49.0 (Kabsch variant RR 63.8% vs 28.2%).
- On ScanNet, FreeReg (Kabsch) reaches RR 78.0% and IN 114.4, versus 42.3%/... for the diffusion-only variant FreeReg-D at RR 42.3%.
- On RGBD-Scenes-v2, FreeReg+PnP reaches RR 57.3% mean, surpassing 2D3D-Matr's 56.4%, though its FMR (82.0%) and IR (30.9%) trail 2D3D-Matr (90.8%, 32.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse diffusion and geometric features instead of using either feature alone (ablation) | FreeReg · 3DMatch | IN (#): 60.8 / 49.4 -> 82.8 | Helped |
| Use ControlNet to extract depth diffusion features rather than directly feeding depth to Stable Diffusion (ablation) | FreeReg · 3DMatch | RR (%): 19.2 -> 33.2 | Helped |
| Set the diffusion layer combination to [0,4,6] rather than a single layer (ablation) | FreeReg diffusion features · 3DMatch | RR (%): 16.7 / 26.0 / 28.7 -> 33.2 | Helped |
| Set the diffusion step to 150 rather than other tested steps (ablation) | FreeReg diffusion features · 3DMatch | RR (%): 31.4 / 31.2 / 31.6 / 30.6 -> 33.2 | Helped |
| Set fusion weight to 0.5 rather than 0.4 (ablation) | FreeReg · 3DMatch | IR (%): 42.9 -> 47.0 | Helped |
| Replace PnP with Kabsch for pose estimation (ablation) | FreeReg · 3DMatch | RR (%): 40.0 -> 63.8 | Helped |
| Replace PnP with Kabsch for pose estimation (ablation) | FreeReg · Kitti-DC | RR (%): 70.5 -> 67.5 | Hurt |
| Use ControlNet rather than directly applying Stable Diffusion to depth maps (ablation) | FreeReg · 3DMatch | IN (#): 30.3 -> 60.8 | Helped |
| Use Dino-v2 features instead of FreeReg diffusion features (ablation) | I2P registration · 3DMatch | IR (%): 47.0 -> 13.0 | Hurt |
| Use CLIP features instead of FreeReg diffusion features (ablation) | I2P registration · 3DMatch | IR (%): 47.0 -> 3.3 | Hurt |
| Use more diffusion layers [0,1,2,6] rather than [0,4,6] (ablation) | FreeReg diffusion features · bundlefusion-office0 | RR (%): 35.9 -> 33.3 | Hurt |
| Add later diffusion layers indexed above 6 (ablation) | FreeReg diffusion features · bundlefusion-office0 | IR (%): 42.3 -> 37.2 / 36.6 | Hurt |
Surprising findings
- Other vision foundation models (DINOv2, CLIP) on depth-projected inputs achieve only up to 13% IR on 3DMatch, far below the diffusion-based features.
- Kabsch is better on indoor datasets but PnP is better on KITTI-DC, because Zoe-Depth has large scale error (3.4m) outdoors.
Limitations
- Slow inference (9.3s per pair) because multiple ControlNet denoising steps are needed.
- Diffusion layers and denoising step were selected manually.
- Kabsch-based pose suffers from Zoe-Depth scale errors, especially outdoors.
Experimental setup
Datasets: 3DMatch ScanNet KITTI-DC RGBD-Scenes-v2 bundlefusion-office0
Benchmarks: 3DMatch ScanNet KITTI-DC RGBD-Scenes-v2
Metrics: Feature Matching Recall Inlier Ratio Inlier Number Registration Recall
Baselines: LCD SuperGlue DeepI2P CN+SG I2P-Matr 2D3D-Matr P2-Net CorrI2P
Builds on: ControlNet Stable Diffusion Zoe-Depth FCGF Kabsch algorithm
Closest papers by meaning
- Cross-PCR: A Robust Cross-Source Point Cloud Registration Framework (AAAI 2025 · similarity 0.90)
- Where Precision Meets Efficiency: Transformation Diffusion Model for Point Cloud Registration (AAAI 2025 · similarity 0.90)
- Implicit Correspondence Learning for Image-to-Point Cloud Registration (CVPR 2025 · similarity 0.89)
- Zero-shot RGB-D Point Cloud Registration with Pre-trained Large Vision Model (CVPR 2025 · similarity 0.89)
- Mining and Transferring Feature-Geometry Coherence for Unsupervised Point Cloud Registration (NeurIPS 2024 · similarity 0.89)
- HybridReg: Robust 3D Point Cloud Registration with Hybrid Motions (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.