Research map / Vision-language and multimodal / Multimodal LLMs
Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Beacon3D, a manually annotated 3D vision-language benchmark for grounding and QA with object-centric metrics and grounding-QA chains, and evaluates grounding/QA coherence of SOTA 3D-VL models.
Problem
Existing 3D vision-language benchmarks have flawed test data (ambiguous grounding text, ambiguous/incomplete QA answers), per-QA averaged metrics vulnerable to visual ignorance and language variation, and isolate grounding from QA, so reported progress may be unreliable.
Approach
Build clean, manually annotated grounding chains (coarse-to-fine referential texts per object) and grounding-QA chains, score per object requiring all three tests to pass, and analyze coherence between grounding and QA outcomes (chain-of-analysis).
Key results
- PQ3D achieves 76.1% per-case but 57.2% object-centric grounding accuracy on Beacon3D ScanNet, an average drop of about 20% when switching to object-centric metrics.
- GPT-4o is the only model with non-trivial object-centric QA accuracy (20.2%), while other models drop to 0.7%-8.1%.
- Human accuracy on the Beacon3D data is about 95%, versus 62% on ScanQA and 69% on ScanRefer validation samples; PQ3D grounding accuracy rose from 63% to 70% after refining ScanRefer texts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Switch from per-case to object-centric evaluation | 3D-VL models on Beacon3D grounding · Beacon3D grounding | accuracy | Hurt |
| Switch from per-case to object-centric evaluation | 3D-VL models on Beacon3D QA · Beacon3D QA | accuracy | Hurt |
| Refine ScanRefer referential texts | PQ3D · ScanRefer val | model accuracy: 63% -> 70% | Helped |
| Replace T5-Small with Vicuna-7B | PQ3D · Beacon3D grounding | object-centric accuracy: 57.2 -> 47.4 | Hurt |
| Replace T5-Small with Vicuna-7B | PQ3D · Beacon3D QA | object-centric accuracy: 3.5 -> 2.3 | Hurt |
| Use object-centric rather than case accuracy to evaluate QA | GPT-4o on Beacon3D · Beacon3D QA | accuracy: 57.1 -> 20.2 | Hurt |
| Train LEO with grounding first, then QA, instead of jointly | LEO · Beacon3D QA | case accuracy: 41.1 -> 43.2 | Helped |
| Use a QA head on a grounding-pretrained, frozen SceneVerse backbone | SceneVerse · Beacon3D QA | case accuracy | Helped |
| Increase grounding-text granularity from coarse to fine | 3D-VL models on grounding chains · Beacon3D grounding chains | grounding accuracy | Hurt |
| Use an LLM for 3D-VL grounding | LLM-based grounding models · Beacon3D grounding | object-centric accuracy | Hurt |
| Use blind LLMs without scene input | SQA3D QA · SQA3D | EM-1: 43.9; 48.8; 49.4; 50.0; 55.6 | Mixed |
| Rephrase questions while preserving the same concept | PQ3D · SQA3D | predictions switching from correct to incorrect: 16% | Hurt |
Surprising findings
- Blind LLMs fine-tuned with LoRA on SQA3D reach accuracy (43.9-50.0 EM-1) close to a state-of-the-art 3D-VL model (55.6), indicating visual ignorance.
- LLM-based 3D-VL models such as Chat-Scene and PQ3D-LLM underperform non-LLM models on grounding, and LLM-based models do not clearly improve object-centric QA.
Limitations
- Object-centric evaluation excludes multi-object grounding and complex reasoning tasks.
- Baseline coverage is limited to a subset of existing 3D-VL models.
- Grounding performance is used as a proxy for implicit grounding in QA, which may be unfair to models with broken grounding implementations (e.g., LEO variants).
Experimental setup
Datasets: ScanNet 3RScan MultiScan ScanRefer ScanQA SQA3D Nr3D Multi3DRefer Sr3D Scan2Cap
Benchmarks: Beacon3D
Metrics: object-centric accuracy per-case accuracy grounding-QA coherence rates R1 and R2 GPT-Score-based accuracy
Baselines: ViL3DRel 3D-VisTA PQ3D SceneVerse GPT-4o LEO-multi LEO-curricular PQ3D-LLM Chat-Scene
Builds on: ScanRefer ScanQA SQA3D LEO PQ3D
Closest papers by meaning
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.91)
- Let Language Constrain Geometry: Vision–Language Models as Semantic and Spatial Critics for 3D Generation (ICML 2026 · similarity 0.90)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.90)
- From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · similarity 0.90)
- ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning (ICML 2026 · similarity 0.90)
- Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.