Research map / 3D vision / 3D generation
TB-HSU: Hierarchical 3D Scene Understanding with Contextual Affordances
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the 3DHSG dataset with region- and object-level affordances over 3RScan scenes and a transformer model (TB-HSU) that jointly classifies room type and region labels to build a three-layer hierarchical scene graph.
Problem
Affordances of objects change with spatial context, but existing 3D scene datasets give fixed or contact-based affordances; the paper targets constructing hierarchical 3D scene graphs (objects, regions, rooms) with context-varying affordances.
Approach
A custom 3DHSG dataset with region-specific and object-specific affordances, and a transformer encoder over object semantic and centroid-position tokens trained with a multi-task loss to jointly predict room type and region labels.
Key results
- On the 3DHSG dataset, TB-HSU with position embedding reaches 91.67% room accuracy and 87.27% region accuracy, above the RF, CNN, and ResNet baselines.
- On Matterport3D, TB-HSU reaches 62.19% room accuracy versus 57.67% for Hydra.
- GPT-4o matches TB-HSU on room classification (91.67%) but reaches only 44.83% region accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add position embedding (ablation) | TB-HSU · 3DHSG room classification | accuracy: 90.28 -> 91.67 | No clear effect |
| add position embedding (ablation) | TB-HSU · 3DHSG region classification | accuracy: 84.95 ± 0.70 -> 87.27 ± 0.98 | Helped |
| add position embedding (ablation) | TB-HSU · ScanNet20 room classification | accuracy: 86.0 -> 86.8 | Helped |
| add position embedding (ablation) | TB-HSU · ScanNet200 room classification | accuracy: 88.6 -> 89.6 | Helped |
| replace learned semantic embedding with pre-trained CLIP text embedding (ablation) | TB-HSU · 3DHSG room classification | mIoU: 74.60 -> 79.63 | Helped |
| replace learned semantic embedding with pre-trained CLIP text embedding (ablation) | TB-HSU · 3DHSG region classification | accuracy: 87.27 ± 0.98 -> 83.86 | Hurt |
| use TB-HSU instead of Hydra | Matterport3D room classification · Matterport3D | accuracy: 57.67 ± 0.57 -> 62.19 ± 0.35 | Helped |
| use TB-HSU with semantic labels and positions instead of PCH with semantic labels | ScanNet20 room classification · ScanNet20 | accuracy: 85.0 -> 86.8 | Helped |
| use TB-HSU with position embedding instead of TB-HSU without position embedding (ablation) | 3DHSG region classification · 3DHSG | mIoU: 74.87 ± 1.23 -> 78.55 ± 2.29 | Helped |
Surprising findings
- GPT-4o performs well on room classification but poorly on region classification (44.83% accuracy), suggesting weak spatial reasoning.
- Position embedding has little effect on room classification but helps region classification.
Limitations
- Room labels for the 3DHSG dataset were obtained with GPT-4o under manual supervision, so GPT-4o's agreement is partly circular.
- Region-level and object-level annotations were refined manually, limiting scale (120 scene graphs).
Experimental setup
Datasets: 3DHSG (custom, built on 3RScan and 3DSSG) ScanNet20 ScanNet200 Matterport3D
Benchmarks: 3DHSG Matterport3D ScanNet20 ScanNet200
Metrics: accuracy mIoU
Baselines: Random Forest with TF-IDF Random Forest with Neighbor-Vote MLP CNN ResNet Hydra PCH GPT-4o
Builds on: 3DSSG 3RScan ViT BERT Hydra
Closest papers by meaning
- SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · similarity 0.91)
- Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces (CVPR 2025 · similarity 0.88)
- 3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds (ICLR 2025 · similarity 0.88)
- GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding (CVPR 2025 · similarity 0.88)
- SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Model (CVPR 2025 · similarity 0.88)
- Grounding 3D Object Affordance with Language Instructions, Visual Observations and Interactions (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.