Research map / Detection and segmentation / 3D detection and driving
Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HCMA, a three-stream hierarchical framework that aligns 3D point clouds, images and text at object, view and scene levels with frozen CLIP encoders to perform open-vocabulary 3D object detection without 3D text annotations.
Problem
Open-vocabulary 3D object detection needs to localize and classify novel objects, but existing VLM-based methods rely on object-centric representations that lose the scene context needed for 3D perception.
Approach
Builds coarse-to-fine 3D-image-text data (object, view, scene levels) via Hierarchical Data Integration, aligns features within and across levels with Interactive Cross-Modal Alignment using contrastive losses, and refines features with an Object-Focusing Context Adjustment module containing Gaussian noise injection.
Key results
- On ScanNet (20 classes), HCMA achieves 21.77 mAP25 versus 19.32 for CoDA and 18.02 for OV-3DET.
- On SUN RGB-D (20 classes), HCMA reports 21.53 mAP25 versus 20.46 for OV-3DET.
- Ablation on ScanNet: full HCMA reaches 20.84 mAP25 / 6.53 mAP50 compared with 19.37 / 5.18 for the baseline without HDI, ICMA, or OFCA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HCMA instead of CoDA | OV-3DOD on ScanNet · ScanNet | mAP25: 19.32 -> 21.77 | Helped |
| Use HCMA instead of OV-3DET | OV-3DOD on SUN RGB-D · SUN RGB-D | mAP25: 20.46 -> 21.53 | Helped |
| Use HCMA instead of FM-OV3D | 10-category detection · ScanNet | mAP25: 21.53 -> 31.63 | Helped |
| Use HCMA instead of FM-OV3D | 10-category detection · SUN RGB-D | mAP25: 21.47 -> 32.35 | Helped |
| Add HDI to the object-level baseline (ablation) | HCMA on ScanNet · ScanNet | mAP25: 19.37 -> 19.71 | Helped |
| Add ICMA after HDI (ablation) | HCMA on ScanNet · ScanNet | mAP25: 19.71 -> 20.18 | Helped |
| Add OFCA after HDI and ICMA (ablation) | HCMA on ScanNet · ScanNet | mAP25: 20.18 -> 20.84 | Helped |
| Add view- and scene-level supervision to object-level supervision (ablation) | HDI on ScanNet · ScanNet | mAP25: 19.37 -> 20.30 | Helped |
| Combine intra-level and inter-level alignment (ablation) | ICMA on ScanNet · ScanNet | mAP25: 19.71 -> 20.18; 19.68 -> 20.18 | Helped |
| Add NIE to OFCA (ablation) | HCMA on ScanNet · ScanNet | mAP25: 20.55 -> 20.84 | Helped |
| Add the top-down image (ablation) | HCMA on ScanNet · ScanNet | mAP25: 20.21 -> 20.30 | Helped |
| Use scene labels to construct alignment pairs (ablation) | HCMA on ScanNet · ScanNet | mAP25: 18.89 -> 19.10 | Helped |
Surprising findings
- HCMA is weaker than the baseline on some classes, e.g., near-zero AP for counter, curtain and scanner, and much lower than OV-3DET on dresser and bookshelf in the tables.
- A fully supervised 3DETR reaches 46.95 mAP25 on ScanNet, far above HCMA, highlighting a large open-vocabulary gap.
Limitations
- Requires a pre-defined vocabulary at inference time.
- Misses transparent objects and some open-vocabulary categories such as curtains.
- Large performance gap to fully supervised 3DETR.
Experimental setup
Datasets: ScanNet SUN RGB-D LVIS ScanNet200
Benchmarks: ScanNet SUN RGB-D ScanNet200
Metrics: AP25 mAP25 mAP50
Baselines: OV-PointCLIP OV-3DETIC OV-CLIP-3D OV-3DET CoDA FM-OV3D OV-3DIC 3DETR
Builds on: 3DETR CLIP OV-3DET Detic
Closest papers by meaning
- FM-OV3D: Foundation Model-Based Cross-Modal Knowledge Blending for Open-Vocabulary 3D Detection (AAAI 2024 · similarity 0.90)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.90)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.88)
- Open-Det: An Efficient Learning Framework for Open-Ended Detection (ICML 2025 · similarity 0.88)
- Cross-Modal 3D Representation with Multi-View Images and Point Clouds (CVPR 2025 · similarity 0.88)
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.