Research map / Detection and segmentation / 3D detection and driving
Towards 3D Objectness Learning in an Open World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OP3Det, a prompt-free, class-agnostic open-world 3D detector that uses SAM and a 2D class-agnostic detector to discover 3D objects with multi-scale point sampling, and trains with a cross-modal mixture-of-experts over point cloud and image features.
Problem
Detecting all objects in 3D scenes, including novel categories unseen in training, without text prompts or class labels; closed-set 3D detectors and open-vocabulary 3D detectors fail to generalize to open-world objectness.
Approach
Use SAM masks prompted by 3D-proximity-based multi-scale point sampling plus a class-agnostic 2D detector to generate extra class-agnostic 3D box supervision, and fuse point cloud, image, and multi-modal features with a cross-modal MoE whose router selects uni-/multi-modal experts.
Key results
- On SUN RGB-D, OP3Det reaches 78.8 AR_novel and 89.7 AR_all, versus 65.3 AR_novel for the best closed-world baseline FCAF3D.
- On ScanNet, OP3Det attains 79.9 AR_novel and 83.2 AR_all, outperforming OV-Uni3DETR's 67.6 AR_novel.
- Cross-dataset SUN RGB-D to ScanNet AR_25 reaches 77.9, a gain of about 30 points over the best baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OP3Det instead of FCAF3D | 3D object detection · SUN RGB-D | AR_novel: 65.3 -> 78.8 | Helped |
| Use OP3Det instead of OV-Uni3DETR | 3D object detection · ScanNet | AR_novel: 67.6 -> 79.9 | Helped |
| Use OP3Det instead of FCAF3D | cross-dataset 3D detection · SUN RGB-D → ScanNet | AR_25: 47.7 -> 77.9 | Helped |
| Use OP3Det instead of Uni3DETR | 3D detection · KITTI | AP_3D medium: 65.74 -> 66.75 | Helped |
| Use a class-specific detector for object discovery | OP3Det · SUN RGB-D class-specific detection | AP_novel: 12.96 -> 14.31 | Helped |
| Add SAM object discovery without point sampling (ablation) | OP3Det · SUN RGB-D | AR_novel: 54.2 -> 50.0 | Hurt |
| Add multi-scale point sampling to SAM object discovery (ablation) | OP3Det · SUN RGB-D | AR_novel: 50.0 -> 69.2 | Helped |
| Add cross-modal MoE (ablation) | OP3Det with SAM and point sampling · SUN RGB-D | AR_novel: 69.2 -> 78.8 | Helped |
| Use image modality only instead of point cloud only (ablation) | 3D object detection · SUN RGB-D | AR_novel: 69.2 -> 38.4 | Hurt |
| Replace CM-MoE with feature addition (ablation) | OP3Det multimodal fusion · SUN RGB-D | AR_novel: 78.8 -> 65.4 | Hurt |
| Replace CM-MoE with feature concatenation (ablation) | OP3Det multimodal fusion · SUN RGB-D | AR_novel: 78.8 -> 66.0 | Hurt |
| Use single-scale point sampling with τ=0.2 instead of SAM alone (ablation) | 3D object discovery · SUN RGB-D discovered objects | AR_novel: 64.0 -> 47.5 | Hurt |
Surprising findings
- Naive fusion by summation or concatenation of point cloud and image features degrades AR relative to point-cloud-only training.
- Using SAM masks alone for 3D object discovery lowers AR_novel and AR_base relative to no discovery, due to fragmented masks.
Limitations
- Misses objects in cluttered scenes and on low-texture, non-rigid, or color-homogeneous regions such as curtains and sofas.
- Relies on accurate 2D-3D projection and limited point cloud coverage.
- AP is not reported for base and novel splits because class-agnostic FP definitions are ambiguous.
Experimental setup
Datasets: SUN RGB-D ScanNet V2 KITTI COCO VOC
Benchmarks: SUN RGB-D ScanNet V2 KITTI COCO
Metrics: AR AP AP_3D AP_BEV F1
Baselines: VoteNet GroupFree FCAF3D Uni3DETR Tr3D CoDA OV-Uni3DETR ImOV3D SECOND PV-RCNN GLRD SAM3D
Builds on: SAM DINO LDET Uni3DETR mmdetection3D
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.91)
- FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection (AAAI 2024 · similarity 0.89)
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships (CVPR 2024 · similarity 0.89)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.89)
- Open-World Objectness Modeling Unifies Novel Object Detection (CVPR 2025 · similarity 0.88)
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.