Research map / Point clouds / Point cloud classification
SGFormer: Semantic Graph Transformer for Point Cloud-Based 3D Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SGFormer, a Transformer-based model for point-cloud 3D scene graph generation that uses edge-aware self-attention (Graph Embedding Layer) and LLM-description cross-attention (Semantic Injection Layer) on 3DSSG.
Problem
Point cloud-based 3D scene graph generation requires modeling global scene structure, which GCN-based methods handle poorly due to over-smoothing and limited neighborhood propagation; long-tail and zero-shot relationships are also hard.
Approach
Replace GCN message passing with Transformer layers using edge-aware self-attention so edge features bias node attention globally, and inject ChatGPT-generated object descriptions (encoded by CLIP) into node features via cross-attention.
Key results
- Relationship R@50 of 56.25 for full SGFormer vs 39.91 for EdgeGCN on 3DSSG (about 40.94% relative improvement).
- Object class R@5 of 96.41 and predicate F1@5 of 91.58 for the full model, the best among compared methods.
- Semantic injection with ChatGPT descriptions plus prompt template improves node mR@5 by 10.21% and edge mR@3 by 22.13% over the baseline without SIL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace EdgeGCN with SGFormer | 3D scene graph generation · 3DSSG test | Relationship R@50: 39.91 -> 56.25 | Helped |
| Replace EdgeGCN with SGFormer | 3D scene graph generation · 3DSSG test | Object R@5: 90.70 -> 96.41 | Helped |
| Add edge features to graph encoding (ablation) | SGFormer with 12 layers · 3DSSG test | Node R@1: 72.53 -> 75.33 | Helped |
| Add edge features to graph encoding (ablation) | SGFormer with 12 layers · 3DSSG test | Edge R@1: 93.67 -> 96.59 | Helped |
| Increase GEL depth from 3 to 12 layers with edge features (ablation) | SGFormer · 3DSSG test | Node R@1: 71.07 -> 75.33 | Helped |
| Increase GEL depth from 3 to 12 layers with edge features (ablation) | SGFormer · 3DSSG test | Edge R@1: 91.58 -> 96.59 | Helped |
| Remove the Semantic Injection Layer (ablation) | SGFormer · 3DSSG test | Node mR@5: 84.16 -> 92.75 | Hurt |
| Remove the Semantic Injection Layer (ablation) | SGFormer · 3DSSG test | Edge mR@3: 53.77 -> 65.67 | Hurt |
| Use category-label text embeddings instead of the full semantic description (ablation) | SGFormer SIL · 3DSSG test | Node mR@5: 84.16 -> 83.67 | Hurt |
| Use category-label text embeddings instead of the full semantic description (ablation) | SGFormer SIL · 3DSSG test | Edge mR@3: 53.77 -> 53.75 | Hurt |
| Use the basic prompt (ablation) | SGFormer SIL · 3DSSG test | Node mR@5: 84.16 -> 88.59 | Helped |
| Use the basic prompt (ablation) | SGFormer SIL · 3DSSG test | Edge mR@3: 53.77 -> 58.50 | Helped |
Surprising findings
- Directly using category-label word embeddings for semantic injection performed worse than the model without SIL.
Limitations
- Improvements on very rare relationships (e.g., plant-standing in-cabinet) remain poor, attributed to limited training data.
- Evaluated only on 3DSSG with ground-truth-like instance segmentation inputs.
Experimental setup
Datasets: 3DSSG 3RScan
Benchmarks: 3DSSG
Metrics: Recall@K macro-F1 mean Recall@K zero-shot Recall@K
Baselines: PointNet SGPN GloRe EdgeGCN
Builds on: EdgeGCN PointNet Transformer CLIP focal loss
Closest papers by meaning
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships (CVPR 2024 · similarity 0.88)
- GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · similarity 0.87)
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.87)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.87)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.