Research map / Vision-language and multimodal / Multimodal LLMs
P$^3$-VLM: A Point-based Alternative for Grounded 3D Vision-Language Models
Anna-Maria Halacheva, Jan-Nico Zaech, Sombit Dey, Luc V Gool, Danda Pani Paudel
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes P^3-VLM, a point-based alternative to bounding boxes for grounded 3D vision-language models, arguing box-based prompts and outputs leak semantic category information and let models ignore visual input.
Problem
Existing 3D VLM grounding benchmarks and prompting protocols rely heavily on bounding boxes, which leak object category semantics and create a shortcut where models neglect visual information during training.
Approach
Replace bounding-box spatial prompts and representation primitives with points, removing the semantic leakage that boxes carry, so that grounding must rely on visual evidence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace box prompt with point prompt | blind LLM · Nr3D | class accuracy (%): 36.4 -> 18.0 | Hurt |
| replace box prompt with point prompt (fine-tuned) | LL3DA · Nr3D | class accuracy (%): 47.7 -> 33.1 | Hurt |
| replace box prompt with point prompt (fine-tuned) | GaussianVLM · Nr3D | class accuracy (%): 46.2 -> 29.3 | Hurt |
| add the location-aware pathway (ablation) | P3-VLM (p) · SQA3D | EM-R: 45.6 -> 51.8 | Improved |
| add the context-aware pathway (ablation) | P3-VLM (p) · ScanQA | EM-R: 22.7 -> 31.9 | Improved |
| add the task-aware pathway (ablation) | P3-VLM (p) · Nr3D | class accuracy (%): 50.2 -> 52.1 | Improved |
| use full tri-pathway design instead of matched baseline (ablation) | P3-VLM (p) · Nr3D | class accuracy (%): 39.8 -> 52.1 | Improved |
| use raw voxel features instead of sparsified features for location pathway (ablation) | P3-VLM (p) · Nr3D | class accuracy (%): 42.3 -> 52.1 | Improved |
| use MLP instead of FiLM for location pathway (ablation) | P3-VLM (p) · Nr3D | class accuracy (%): 48.5 -> 52.1 | Improved |
| use 3D RoPE instead of learned absolute PE in sparsifier (ablation) | P3-VLM (p) · Nr3D | class accuracy (%): 49.4 -> 52.1 | Improved |
| train on random surface points instead of object centers | P3-VLM (p) · Nr3D | class accuracy (%): 52.1 -> 58.8 | Improved |
| point prompt instead of box prompt in training | P3-VLM · GaussianVLM-OOD | EM: 15.42 -> 17.31 | Improved |
Surprising findings
- A 'blind' language-only model can perform well on existing 3D grounding benchmarks because bounding-box prompts leak object category information.
Closest papers by meaning
- Grounded 3D-Aware Spatial Vision-Language Modeling (CVPR 2026 · similarity 0.92)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.91)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.91)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.90)
- MonoVLM: Monocular 3D Visual Grounding with Vision Language Models (CVPR 2026 · similarity 0.90)
- ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
