Research map / Point clouds / Point cloud classification
Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OccTIP, an occlusion-aware text-image-point cloud pretraining framework built from synthetic ShapeNetCore models, and DuoMamba, a two-stream Mamba-based point cloud encoder with Hilbert curves and standard 1D convolutions, for open-world 3D recognition.
Problem
Zero-shot 3D object recognition on real, occluded point clouds lags because CLIP-aligned 3D encoders are pretrained on complete synthetic point clouds, and Transformer-based encoders have high inference cost.
Approach
Generate partial (single-view, visible-only) point clouds from synthetic 3D meshes rendered from 12 camera positions, align them with frozen CLIP text and image encoders via cross-modal contrastive learning, and use a linear-time two-stream Mamba block that orders point patches along Hilbert and Trans-Hilbert curves and uses standard (non-causal) 1D convolutions.
Key results
- OccTIP with DuoMamba reaches 63.5% zero-shot top-1 accuracy on ScanObjectNN, and OccTIP lifts PointBERT and SparseConv zero-shot ScanObjectNN accuracy by 5.1% and 3.8%.
- On ScanNetV2 zero-shot 3D detection, OccTIP achieves mAP25 of 28.9% and mAP50 of 22.7%.
- DuoMamba reaches 63.5% on ScanObjectNN versus 60.7% for Mamba3D and 62.6% for PointMamba, with 7.1G FLOPs versus 10.3G for PointMamba.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain SparseConv on occlusion-aware partial point clouds instead of the existing pretraining setup | SparseConv · ScanObjectNN | Top 1 accuracy: 57.9 -> 61.7 | Helped |
| pretrain PointBERT on occlusion-aware partial point clouds instead of the existing pretraining setup | PointBERT · ScanObjectNN | Top 1 accuracy: 55.5 -> 60.6 | Helped |
| use DuoMamba with OccTIP pretraining instead of PointBERT with OccTIP pretraining | OccTIP-pretrained 3D encoder · ScanObjectNN | Top 1 accuracy: 60.6 -> 63.5 | Helped |
| use DuoMamba with OccTIP pretraining instead of PointBERT with OccTIP pretraining | OccTIP-pretrained 3D encoder · ScanObjectNN | 16-shot accuracy: 76.8 -> 79.4 | Helped |
| use DuoMamba with OccTIP pretraining instead of PointBERT with OccTIP pretraining | OccTIP-pretrained 3D encoder · ScanNetV2 instances | class-average accuracy: 47.8 -> 49.0 | Helped |
| pretrain with OccTIP instead of OpenShape | SparseConv · ScanNetV2 instances | class-average accuracy: 39.9 -> 45.3 | Helped |
| pretrain with OccTIP instead of MixCon3D | 3D detector classification encoder · ScanNetV2 | mAP25: 24.1 -> 28.9 | Helped |
| pretrain with OccTIP instead of TAMM | 3D detector classification encoder · SUN RGB-D | mAP25: 18.9 -> 24.4 | Helped |
| add Hilbert and Trans-Hilbert sorting and standard Conv1D to the FPS-ordered baseline (ablation) | DuoMamba · ScanObjectNN | accuracy: 60.6 -> 63.5 | Helped |
| add standard Conv1D (ablation) | DuoMamba with Hilbert and Trans-Hilbert sorting · ScanObjectNN | accuracy: 63.1 -> 63.5 | Helped |
| use Hilbert sorting rather than Trans-Hilbert sorting alone, retaining Conv1D (ablation) | DuoMamba · ScanObjectNN | accuracy: 61.7 -> 62.2 | Helped |
| replace FPS order with Hilbert and Trans-Hilbert curves (ablation) | DuoMamba · ScanObjectNN | accuracy: 61.8 -> 63.5 | Helped |
Limitations
- Did not pretrain on Objaverse (about 800K objects) due to resource constraints.
Experimental setup
Datasets: ShapeNetCore ScanObjectNN ModelNet40 ScanNetV2 SUN RGB-D LAION-5B
Benchmarks: ModelNet40-P ScanObjectNN ScanNetV2 SUN RGB-D
Metrics: top-1 accuracy top-3 accuracy top-5 accuracy few-shot accuracy mAP25 mAP50
Baselines: OpenShape MixCon3D TAMM OpenDlign CLIP2 PointCLIP PointCLIP V2 Mamba3D PointMamba
Builds on: CLIP Mamba OpenShape MixCon3D PointBERT
Closest papers by meaning
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.90)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images (NeurIPS 2024 · similarity 0.88)
- Duoduo CLIP: Efficient 3D Understanding with Multi-View Images (ICLR 2025 · similarity 0.88)
- Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysis (CVPR 2025 · similarity 0.88)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.