Research map / Point clouds / Point cloud classification
ObjVariantEnsemble: Advancing Point Cloud LLM Evaluation in Challenging Scenes with Subtly Distinguished Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds ObjVariantEnsemble (OVE), a benchmark of ~75k synthetic-by-assembly 3D scenes that insert subtly distinguished object distractors into ScanNet backgrounds, annotated by an LLM-VLM pipeline, and evaluates 3D grounding models on it.
Problem
Existing 3D scene benchmarks built on ScanNet are small, have ambiguous text annotations, and lack customizable challenge levels, so the capabilities of point cloud LLMs and 3D grounding models in cluttered scenes with similar objects are under-evaluated.
Approach
Assemble new scenes by inserting object-level scanned or CAD objects (from OmniObject, ScanObjectNN, ModelNet, ShapeNet) that resemble the target into real scene backgrounds, then generate fine-grained difference annotations with an iterative LLM-VLM questioning loop (LLaVA plus GPT summarization) and spatial predicates.
Key results
- Chat-3D-v2 performs much worse on OVE than its reported ScanRefer results, while a fine-tuned version using part of OVE improves.
- M3DRef-CLIP matches ScanRefer average performance with 2 distractors but declines as distractor count grows from 2 to 10.
- Models do far worse on location-only distinctions than on shape, color, or class distinctions, indicating weak pure spatial reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add distractor objects to the scene | PointBERT and Uni3D · OVE object partial segmentation | mIoU_I | Hurt |
| Train on scenes containing objects from the same class instead of different classes | 3D segmentation models · OVE object partial segmentation | mIoU_I | Helped |
| Distinguish targets using shape or class rather than location alone | M3DRef-CLIP and fine-tuned Chat-3D-v2 · OVE 3D grounding | Acc@0.25 | Helped |
| Increase the number of distractors | M3DRef-CLIP and Chat-3D-v2 · OVE 3D grounding | Acc@0.25 | Hurt |
| Fine-tune Chat-3D-v2 using a portion of OVE | Chat-3D-v2 · OVE 3D grounding | Acc@0.25 | Helped |
Surprising findings
- The original Chat-3D-v2 performs much worse on OVE than on ScanRefer, possibly due to overfitting to ScanRefer.
- Chat-3D-v2's extra location encoding does not surpass Multi3DRef in challenging scenarios.
Limitations
- Object-level datasets retain only point clouds without mesh or texture, so OVE cannot render rich 2D images.
- Shape coverage limited to cuboids, L-shapes and spheres.
- Annotation relies on LLM/VLM outputs with only sampled manual verification.
Experimental setup
Datasets: ScanNet ScanRefer OmniObject ScanObjectNN ModelNet ShapeNet
Benchmarks: ObjVariantEnsemble (OVE)
Metrics: Acc@0.25 Acc@0.5 mIoU
Baselines: Chat-3D-v2 Multi3DRef (M3DRef-CLIP) ScanRefer
Builds on: ScanRefer Multi3DRef Chat-3D-v2 LLaVA Uni3D
Closest papers by meaning
- Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships (CVPR 2024 · similarity 0.87)
- Point2Real: Bridging the Gap between Point Cloud and Realistic Image for Open-World 3D Recognition (AAAI 2024 · similarity 0.86)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.86)
- Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition (CVPR 2025 · similarity 0.86)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.86)
- Interpretable3D: An Ad-Hoc Interpretable Classifier for 3D Point Clouds (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.