Research map / Point clouds / Registration and matching
Zero-shot RGB-D Point Cloud Registration with Pre-trained Large Vision Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZeroMatch, a training-free RGB-D point cloud registration framework that fuses FPFH geometric descriptors with Stable Diffusion UNet features computed on a coupled source-target image with consistency attention.
Problem
Zero-shot RGB-D point cloud registration: achieving robust rigid registration on unseen scenes without any task-specific training, where FPFH features are ambiguous in repetitive or low-overlap scenes and learned RGB-D methods generalize poorly.
Approach
Concatenates FPFH local geometric descriptors with training-free Stable Diffusion UNet features extracted from a vertically stacked source-target image, using modified self-attention and a coupled prompt so features are cross-view consistent, then estimates pose via geometric-compatibility sample consensus.
Key results
- On ScanNet, ZeroMatch reaches 98.9% rotation accuracy at 5 degrees and 89.9% translation accuracy at 5 cm, exceeding the trained NeRF-UR (97.8% and 86.9%).
- On ScanLoNet, ZeroMatch gains up to 4.5% in rotation accuracy at 5 degrees and 6% in translation accuracy at 5 cm over prior deep RGB-D methods.
- On 3DMatch-derived RGB-D pairs (40-frame interval), ZeroMatch achieves 93.5% rotation accuracy at 5 degrees versus 80.9% for PointMBF trained on 3DMatch.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use coupled-image input instead of single-image input (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 66.5 -> 67.5 | Helped |
| add prompt-to-image coupled consistency attention using the coupled prompt (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 66.1 -> 67.5 | Helped |
| add image-level coupled consistency attention (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.4 -> 67.5 | Helped |
| set ε to 0.6 instead of 0.8 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 67.1 | Hurt |
| set ε to 0.7 instead of 0.8 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 67.1 | Hurt |
| set ε to 0.9 instead of 0.8 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 67.3 | Hurt |
| set ε to 1.0 instead of 0.8 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 67.4 | Hurt |
| combine geometric and Stable Diffusion features (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 60.1 -> 67.5; 63.0 -> 67.5 | Helped |
| compress Stable Diffusion features to dimension 64 instead of 128 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 66.7 | Hurt |
| compress Stable Diffusion features to dimension 256 instead of 128 (ablation) | ZeroMatch · ScanLoNet | accuracy @ 5: 67.5 -> 67.6 | No clear effect |
Surprising findings
- A training-free pipeline with frozen Stable Diffusion features outperforms learned RGB-D registration methods trained on 3DMatch or ScanNet, including NeRF-UR, on the reported benchmarks.
Limitations
- Relies on handcrafted FPFH for local geometry, so accuracy depends on the handcrafted descriptor's quality.
- Evaluation uses self-constructed RGB-D pairs for 3DMatch and ScanLoNet, making direct comparison with original benchmarks difficult.
- Baseline numbers on some datasets are taken from prior papers (e.g., NeRF-UR code unavailable).
Experimental setup
Datasets: ScanNet 3DMatch ScanLoNet
Benchmarks: ScanNet ScanLoNet 3DMatch
Metrics: rotation accuracy at 5/10/45 degrees translation accuracy at 5/10/25 cm Chamfer distance accuracy mean and median rotation, translation, and Chamfer errors
Baselines: ICP SIFT SuperPoint FCGF DGR 3D MV Reg REGTR GeoTransformer PEAL UR&R BYOC LLT
Builds on: Stable Diffusion FPFH SC2-PCR UR&R PointMBF
Closest papers by meaning
- FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators (ICLR 2024 · similarity 0.89)
- Generative Point Cloud Registration (ICML 2025 · similarity 0.88)
- Cross-PCR: A Robust Cross-Source Point Cloud Registration Framework (AAAI 2025 · similarity 0.87)
- Implicit Correspondence Learning for Image-to-Point Cloud Registration (CVPR 2025 · similarity 0.87)
- A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration (NeurIPS 2024 · similarity 0.86)
- 3D Focusing-and-Matching Network for Multi-Instance Point Cloud Registration (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.