Research map / Point clouds / Point cloud classification
Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Open3DSG distills 2D vision-language features (OpenSeg for objects, InstructBLIP for relationships) into a 3D point-cloud GNN, enabling zero-shot open-vocabulary 3D scene graph prediction with an LLM producing open-set relationships.
Problem
3D scene graph prediction is limited to fixed object and relationship label sets from small labeled datasets, and 2D vision-language models handle compositional relationships poorly.
Approach
Co-embed a 3D point-cloud GNN's node and edge features with 2D VLM feature spaces (OpenSeg for objects, InstructBLIP BLIP encoder for relation pairs) via cosine loss, then query objects with CLIP text embeddings and generate relationships with InstructBLIP's Qformer and LLM conditioned on the 3D graph features.
Key results
- On 3DSSG, Open3DSG reaches object R@5 0.57 and predicate R@3 0.63, comparable to the fully supervised 3DSSG baseline (0.68 and 0.89 respectively)
- On tail object classes, Open3DSG achieves 0.42 R@5 versus 0.06 for 3DSSG and 0.24 for SGRec3D
- Naive CLIP relationship prediction achieves only 0.04 R@100, showing CLIP-like models are ill-suited for relationships
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Open3DSG instead of the strongest fully supervised baseline | 3D scene graph prediction · 3DSSG | object R@5: 0.78 -> 0.57 | Hurt |
| use Open3DSG instead of the strongest fully supervised baseline | 3D scene graph prediction · 3DSSG | predicate R@3: 0.99 -> 0.63 | Hurt |
| use Open3DSG instead of the strongest fully supervised baseline | 3D scene graph prediction · 3DSSG | relationship R@50: 0.90 -> 0.64 | Hurt |
| use fused 2D and 3D features (ablation) | Open3DSG · 3DSSG | object R@5: 0.46 -> 0.57 | Helped |
| use fused 2D and 3D features (ablation) | Open3DSG · 3DSSG | predicate R@3: 0.60 -> 0.63 | Helped |
| replace OpenSeg with CLIP for object features (ablation) | Open3DSG · 3DSSG | object R@5: 0.57 -> 0.48 | Hurt |
| provide ground-truth object labels (ablation) | Open3DSG relationship prediction · 3DSSG | predicate R@3: 0.63 -> 0.64 | No clear effect |
| fine-tune with supervised relationship labels (ablation) | Open3DSG · 3DSSG | predicate R@3: 0.63 -> 0.76 | Helped |
| fine-tune with supervised relationship labels (ablation) | Open3DSG · 3DSSG | object R@5: 0.57 -> 0.59 | Helped |
Surprising findings
- A naive CLIP-based relationship prediction performs close to chance (0.04 R@100), while the LLM-based generative approach performs far better
- Using CLIP instead of OpenSeg for object features lowers object R@5 from 0.57 to 0.48
Limitations
- Open-vocabulary relationship prediction remains challenging and lags fully supervised methods on closed-set metrics
- Evaluation of open-vocabulary scene graphs relies on closed-vocabulary metrics and label mapping via BERT
- Gap to state-of-the-art fully supervised methods on 3DSSG
Experimental setup
Datasets: ScanNet 3DSSG
Benchmarks: 3DSSG
Metrics: R@k mR@k
Baselines: 3DSSG SGFN SGRec3D VL-SAT CLIP (naive) NegCLIP ConceptGraphs-style caption baseline
Builds on: 3DSSG OpenSeg InstructBLIP CLIP OpenScene
Closest papers by meaning
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.91)
- Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces (CVPR 2025 · similarity 0.90)
- OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations (NeurIPS 2025 · similarity 0.90)
- Towards 3D Objectness Learning in an Open World (NeurIPS 2025 · similarity 0.89)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.88)
- CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.