Research map / Vision-language and multimodal / Multimodal LLMs
From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Anywhere3D-Bench, a 2,886-pair multi-level 3D visual grounding benchmark (area, space, object, part) and evaluates LLMs, MLLMs, and 3D VG specialist models, finding space- and part-level grounding hardest.
Problem
Existing 3D visual grounding benchmarks focus on objects, leaving grounding of referring expressions to regions beyond objects (activity areas, unoccupied space) and object parts largely unexplored, and it is unclear whether LLMs/MLLMs can predict precise 3D boxes for such expressions.
Approach
A hierarchical four-level benchmark (area, space, object, part) with GPT-4o-generated expressions from scene graphs, human-in-the-loop revision, and bounding-box annotation, plus a level-specific IoU metric; models are given ground-truth object boxes and captions so evaluation isolates reasoning rather than detection.
Key results
- Gemini-2.5-Pro achieves 46.47% overall and 29.86% at space level; o3 reaches 31.26% at space level and 38.77% at part level.
- Human evaluation on a 200-expression subset reaches 95% overall versus 92% at space level.
- Thinking models consistently outperform non-thinking counterparts; the best 3D VG specialist (Chat-Scene) reaches only 22.90% overall.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video frames and a bird's-eye view image as visual inputs | GPT-4.1, o4-mini, and Qwen2.5-VL-72B · Anywhere3D-Bench, object level | Acc@0.25IoU: 8.19% on average | Helped |
| add video frames and a bird's-eye view image as visual inputs | GPT-4.1, o4-mini, and Qwen2.5-VL-72B · Anywhere3D-Bench, part level | Acc@0.25IoU: 4.76% on average | Helped |
| add video frames and a bird's-eye view image as visual inputs | GPT-4.1, o4-mini, and Qwen2.5-VL-72B · Anywhere3D-Bench, space level | Acc@0.25IoU: 3.47% on average | Helped |
| enable thinking | Qwen3-32B, DeepSeek, and Gemini-2.0-Flash · Anywhere3D-Bench, all four grounding levels | Acc@0.25IoU: nearly 9% | Helped |
| use a left-handed rather than right-handed coordinate system | LLMs and MLLMs evaluated on directional-term mapping · left-handed coordinate system | accuracy | Hurt |
| use cardinal directions instead of coordinate axes | LLMs and MLLMs evaluated on directional-term mapping · cardinal-direction mapping | accuracy | Helped |
Surprising findings
- Models perform far worse with a left-handed coordinate system, and even reasoning models make errors mapping left/right to axes, while cardinal-direction prompts are handled much better.
- Specialist 3D visual grounding models underperform MLLMs even with ground-truth object masks, with Chat-Scene reaching only 31.73% on object-level grounding versus over 50% on ScanRefer.
Limitations
- Models are given ground-truth object boxes and captions, so detection errors are excluded from evaluation.
- Expressions are generated by GPT-4o and then human-revised, which may introduce generator-specific style biases.
- 3D VG specialists are evaluated only on the ScanNet portion for some models.
Experimental setup
Datasets: ScanNet MultiScan 3RScan ARKitScenes
Benchmarks: Anywhere3D-Bench
Metrics: Acc@0.25IoU
Baselines: 3D-VisTA PQ3D Chat-Scene Grounded 3D-LLM GPT4Scene LLaVA-OneVision Qwen2.5-VL-72B InternVL3-8B
Builds on: ScanRefer GPT4Scene SceneVerse SQA3D
Closest papers by meaning
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.93)
- CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding (ICLR 2024 · similarity 0.91)
- 3EED: Ground Everything Everywhere in 3D (NeurIPS 2025 · similarity 0.91)
- Viewpoint-Aware Visual Grounding in 3D Scenes (CVPR 2024 · similarity 0.90)
- Multi-Object 3D Grounding with Dynamic Modules and Language-Informed Spatial Attention (NeurIPS 2024 · similarity 0.90)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.