Research map / Point clouds / Point cloud classification
CLIP-Driven Open-Vocabulary 3D Scene Graph Generation via Cross-Modality Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CCL-3DSGG, which trains a 3D scene graph feature extractor without 3DSG labels by contrastively aligning CLIP text (grammar-parsed words with adjective-swap negatives) and multi-view image features to 3D scene graph features, enabling open-vocabulary and zero-shot 3DSGG.
Problem
3D scene graph generation requires costly ground-truth object and predicate annotations and is restricted to closed-set classes, so it cannot recognize novel objects and predicates.
Approach
Align word-level text parsed by grammar parsing (with adjective-exchange negatives) and multi-view CLIP image features to 3DSG features via contrastive losses on naturally paired captions and images, then perform open-vocabulary inference by cosine similarity between CLIP prompts and 3DSG features.
Key results
- Supervised SGCLS/PREDCLS mean Recall 60.1 and mean mR 51.3 on 3DSSG, surpassing VL-SAT (54.4/47.1).
- Unsupervised mR@100 PREDCLS of 49.1 vs VL-SAT w/o CL 27.4 on 3DSSG.
- Zero-shot 3DSGG PREDCLS R@100 65.7 vs VL-SAT 59.4 (R@50 49.1 vs 43.5 on SGCLS/PREDCLS table).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CCL-3DSGG instead of VL-SAT | 3DSGG on 3DSSG · 3DSSG SGCLS | R@100: 33.7 -> 45.7 | Helped |
| Use CCL-3DSGG instead of VL-SAT | 3DSGG on 3DSSG · 3DSSG unsupervised PREDCLS | mR@100: 27.4 -> 49.1 | Helped |
| Use CCL-3DSGG instead of VL-SAT | 3DSGG on 3DSSG · 3DSSG open-vocabulary SGCLS | R@100: 29.4 -> 42.3 | Helped |
| Add word-level grammar parsing rather than using sentence-level features (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 6.8 -> 29.4 | Helped |
| Add adjective exchange for negative text samples (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 25.8 -> 29.4 | Helped |
| Add the multi-view image-3DSG contrastive loss (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 13.8 -> 29.4 | Helped |
| Add the text-3DSG contrastive loss (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 5.3 -> 29.4 | Helped |
| Add image features during inference (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 24.3 -> 29.4 | Helped |
| Replace the designed prompt with a learnable prompt (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 29.4 -> 25.3 | Hurt |
| Add object classification loss (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 29.4 -> 31.4 | Helped |
| Add predicate classification loss (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 29.4 -> 50.5 | Helped |
| Use the VL-SAT prediction head (ablation) | CCL-3DSGG · 3DSSG PREDCLS | mR@20: 29.4 -> 28.6 | Hurt |
Surprising findings
- Removing the text-3DSG contrastive loss causes the largest unsupervised drop (mR@20 from 29.4 to 5.3), more than removing the image loss (to 13.8).
Limitations
- Image features at test time are fused late, so pixel-level information is not fully exploited.
- Open-vocabulary quality on ScanNet is only qualitatively evaluated due to missing ground truth.
Experimental setup
Datasets: 3DSSG ScanRefer ScanNet
Benchmarks: 3DSSG ScanNet
Metrics: Recall@K mean Recall@K mA@k top-k accuracy
Baselines: SGPN SGFN EdgeGCN KISGP VL-SAT Chen et al. (CGF2022) Feng et al. (CVPR2023)
Builds on: VL-SAT CLIP PointNet ULIP
Closest papers by meaning
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships (CVPR 2024 · similarity 0.88)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.87)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.87)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.87)
- RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · similarity 0.87)
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.