Research map / Detection and segmentation / Open-set and few-shot detection
OpenLex3D: A Tiered Benchmark for Open-Vocabulary 3D Scene Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OpenLex3D, a tiered human-annotated benchmark relabeling Replica, ScanNet++ and HM3D scenes with synonym, depiction, visually-similar and clutter labels, and evaluates open-vocabulary 3D scene representations on open-set segmentation and object retrieval.
Problem
Open-vocabulary 3D scene representations are evaluated with closed-set semantic labels and metrics, which fail to capture the variability of natural language labels and do not diagnose specific failure modes.
Approach
Each object receives multiple human-annotated free-form noun labels organized into four precision tiers (synonyms, depictions, visually similar, clutter), enabling tiered segmentation metrics (Top-N Frequency, Set Ranking) and procedurally generated retrieval queries covering all objects.
Key results
- ConceptGraphs (GPT) achieves the highest top-5 synonym frequency on the benchmark, while OpenScene and Kassab2024 perform best on depiction and visually similar categories.
- Object retrieval AP is low overall: OpenMask3D+NMS reaches 11.47 mAP on Replica, but ConceptGraphs leads on HM3D with 5.09 mAP.
- The OpenLex3D label sets provide 13 times more labels per scene than the original datasets and cover 3812 objects labeled by four annotators each.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-generated object captions instead of standard ConceptGraphs features | ConceptGraphs · Replica | R_S | Helped |
| Use GPT-generated object captions instead of standard ConceptGraphs features | ConceptGraphs · ScanNet++ | R_S | Helped |
| Use GPT-generated object captions instead of standard ConceptGraphs features | ConceptGraphs · HM3D | R_S | Helped |
| Use GPT-generated object captions instead of standard ConceptGraphs features | ConceptGraphs · all datasets | mR | Hurt |
| Include depiction information in retrieval queries in addition to synonyms | object retrieval methods · Replica, ScanNet++, and HM3D | AP | Helped |
| Use OpenMask3D with NMS and clean mesh inputs | object retrieval · Replica and ScanNet++ | mAP | Helped |
| Use OpenMask3D with NMS and clean mesh inputs | object retrieval · HM3D | mAP | Hurt |
| Expand the Replica prompt list from 101 to 759 labels with plausible imprecise labels | semantic segmentation methods · Replica | mIoU | Hurt |
Surprising findings
- Adding plausible but imprecise labels (depictions and visually similar) to Replica prompt lists markedly reduces mIoU for all tested methods.
- Retrieval performance is substantially higher for depiction-augmented queries than for synonym-only queries, and ConceptGraphs (GPT) benefits least.
Limitations
- Label set does not cover affordances, material, or color, and omits small object parts beyond original segments.
- Object retrieval queries are generated from a simple template and inherit the same limitations.
- Standard deviations across metrics are non-negligible, and OpenMask3D's clutter results vary strongly due to fixed-class Mask3D.
Experimental setup
Datasets: Replica ScanNet++ HM3D
Benchmarks: OpenLex3D
Metrics: Top-N Frequency Set Ranking (mR, R_S, R_DVS, penalties) mAP AP50 AP25 mIoU
Baselines: ConceptGraphs ConceptGraphs (GPT) HOV-SG Kassab2024 OpenMask3D ConceptFusion OpenScene
Builds on: SimLex-999 OpenMask3D ConceptGraphs OpenScene OpenSun3D
Closest papers by meaning
- Open-Vocabulary 3D Semantic Segmentation with Foundation Models (CVPR 2024 · similarity 0.91)
- Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · similarity 0.90)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.90)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
- Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding (CVPR 2024 · similarity 0.90)
- Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.