Research map / Large language models / Agents, code and math
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces 3D-GRAND, a 6.2M-pair densely grounded 3D-scene/text instruction dataset built from synthetic indoor scenes with GPT-4/GPT-4V annotation, and 3D-POPE, a hallucination benchmark for 3D-LLMs; a LoRA-tuned Llama-2 trained on it improves ScanRefer grounding and POPE scores.
Problem
3D-LLMs lack large-scale, densely grounded 3D scene-language data, and their object hallucination has not been systematically measured.
Approach
Scale dense noun-level grounding of language to 3D objects by using LLM/VLM annotation over scene graphs built from synthetic 3D-FRONT and Structured3D scenes, filter hallucinated object IDs, and evaluate hallucination with a POPE-style yes/no existence benchmark in 3D.
Key results
- On ScanRefer, the 3D-GRAND model reaches 38.0% Acc@0.25 zero-shot (never seen ScanNet), beating 3D-LLM's 30.3%.
- On 3D-POPE random split, 3D-GRAND achieves 93.34% precision and 89.12% accuracy, far above 3D-LLM, LEO and 3D-VisTA.
- GPT-4-based densely grounded annotation shows 5.6-8.2% error rate, comparable to professional annotators (<5%) and better than AMT (16%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on 3D-GRAND instead of the reported 3D-LLM baseline training | 3D-LLM · ScanRefer validation | Accuracy@0.25: 30.3 -> 38.0 | Helped |
| train on synthetic 3D-GRAND scenes without ScanNet scenes | 3D-LLM · ScanRefer validation | Accuracy@0.25: 35.2 -> 38.0 | Helped |
| use 3D-GRAND 0-shot grounding model rather than LEO | 3D-POPE · Random | F1 score: 62.25 -> 88.56 | Helped |
| use 3D-GRAND 0-shot grounding model rather than LEO | 3D-POPE · Popular | F1 score: 59.55 -> 78.26 | Helped |
| use 3D-GRAND 0-shot grounding model rather than LEO | 3D-POPE · Adversarial | F1 score: 59.78 -> 76.37 | Helped |
| add dense grounding tokens (ablation) | 3D-GRAND · 3D-POPE Random | Precision: +1.38 | Helped |
| add dense grounding tokens (ablation) | 3D-GRAND · 3D-POPE Popular | Precision: +2.68 | Helped |
| add dense grounding tokens (ablation) | 3D-GRAND · 3D-POPE Adversarial | Precision: +2.38 | Helped |
| ground each mentioned object before giving the final answer rather than afterward (ablation) | 3D-GRAND · ScanRefer validation, Mask3D Top40 | Accuracy@0.25 overall: 36.3 -> 38.0 | Helped |
| add dense grounding to the model (ablation) | 3D-GRAND · ScanRefer validation, Mask3D Top40 | Accuracy@0.25 overall: 34.2 -> 38.0 | Helped |
| add dense grounding to the model (ablation) | 3D-GRAND · ScanRefer validation, GT boxes | Accuracy@0.25 overall: 53.4 -> 57.0 | Helped |
| scale up the amount of densely grounded training data (ablation) | 3D-GRAND · ScanRefer and 3D-POPE | grounding accuracy and hallucination rate | Helped |
Surprising findings
- Zero-shot sim-to-real transfer: a model trained only on synthetic scenes outperforms ScanNet-trained 3D-LLM on ScanRefer Acc@0.25.
Limitations
- Model is primarily text-based with object-centric scene graph input, conveying little visual information.
- Performance drops substantially on harder 3D-POPE Popular and Adversarial splits.
- Annotation quality relies on GPT-4 and GPT-4V with residual error rates of 5.6-8.2%.
Experimental setup
Datasets: 3D-FRONT Structured3D ScanNet ScanRefer ScanNet200
Benchmarks: 3D-POPE ScanRefer
Metrics: Precision Recall F1 Score Accuracy Yes (%) Acc@0.25IoU Acc@0.5IoU
Baselines: 3D-LLM LEO 3D-VisTA SceneVerse LLM-Grounder M3DRef-CLIP ViL3DRel 3DVG-Trans MVT ScanRefer
Builds on: POPE LEO 3D-LLM Set-of-Mark prompting
Closest papers by meaning
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (ICML 2024 · similarity 0.89)
- Grounded Chain-of-Thought for Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.88)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.88)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.88)
- MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.