Research map / Vision-language and multimodal / Visual question answering
3D Question Answering via only 2D Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cdViews, a zero-shot 3D-QA method that feeds a frozen LLaVA-OV-7B with a small set of critical and diverse 2D views selected by a lightweight viewSelector and a camera-pose-based viewNMS.
Problem
Zero-shot 3D question answering using only 2D vision-language models, where the limited token budget makes selecting which rendered/sampled views to feed the LVLM critical.
Approach
Select views by a learned criticality score (trained on automatically annotated positive/negative views from a frozen LVLM given caption rephrasings of QA pairs) and then remove spatially redundant views using camera position and orientation distances via a view NMS, while keeping the LVLM frozen.
Key results
- On ScanQA test set, LLaVA-OV + cdViews reaches 35.0 EM@1 (w/ object) and 102.8 CIDEr, versus 31.3 EM@1 and 83.8 CIDEr for BridgeQA.
- On SQA test set, LLaVA-OV + cdViews reaches 56.8 EM@1, versus 52.9 for BridgeQA.
- viewNMS reduces the number of input views from 17 to 9 while raising ScanQA validation EM@1 from 29.7 to 30.1 with the full cdViews pipeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform view sampling with cdViews | LLAVA-OV-7B · ScanQA test with objects | EM@1: 33.1 -> 35.0 | Helped |
| Replace image-text retrieval with cdViews | LLAVA-OV-7B · SQA test | EM@1: 55.0 -> 56.8 | Helped |
| Use cdViews instead of the compared best method | LLAVA-OV-7B · SQA test, Is questions | EM@1: -2.8 | Hurt |
| Use cdViews instead of the compared best method | LLAVA-OV-7B · SQA test, Can questions | EM@1: -10.2 | Hurt |
| Replace uniform sampling with image-text retrieval (ablation) | LLAVA-OV · ScanQA validation | EM@1: 28.3 -> 29.1 | Helped |
| Add viewNMS to image-retrieval view selection (ablation) | LLAVA-OV + retrieval · ScanQA validation | EM@1: 29.1 -> 29.2 | Helped |
| Add viewSelector to uniform view selection (ablation) | LLAVA-OV · ScanQA validation | EM@1: 28.3 -> 29.7 | Helped |
| Add viewNMS to viewSelector-selected views (ablation) | LLAVA-OV + viewSelector · ScanQA validation | EM@1: 29.7 -> 30.1 | Helped |
| Replace retrieval plus viewNMS with viewSelector plus viewNMS (ablation) | LLAVA-OV · ScanQA validation | EM@1: 29.2 -> 30.1 | Helped |
| Set the viewNMS threshold to 0.5 (ablation) | LLAVA-OV + cdViews · ScanQA validation | EM@1 | Helped |
| Increase the viewNMS threshold beyond 0.5 with fixed k=9 (ablation) | LLAVA-OV + cdViews · ScanQA validation | EM@1 | Hurt |
| Replace uniform sampling with cdViews (ablation) | LLAVA-Next · ScanQA validation | EM@1: 21.0 -> 24.6 | Helped |
Surprising findings
- The zero-shot 2D-only method underperforms on SQA 'Is' and 'Can' questions relative to dataset-adapted 3D methods (drops of 2.8 and 10.2 points).
- Image retrieval gives only a modest gain over uniform sampling (29.1 vs 28.3 EM@1 on ScanQA validation).
Limitations
- Performance on SQA 'Is' and 'Can' questions lags dataset-adapted baselines.
- viewSelector requires auto-annotated training data generated by the LVLM itself and a tuned NMS threshold and view count k selected on validation.
Experimental setup
Datasets: ScanQA SQA
Benchmarks: ScanQA SQA
Metrics: EM@1 BLEU-1 ROUGE-L CIDEr
Baselines: ScanQA SQA3D 3D-LLM 3D-VLP 3D-VisTA SIG3D SynFormer3D LL3DA PQ3D BridgeQA GPT-4O+CC LLaVA-OV uniform sampling
Builds on: LLaVA-OneVision BridgeQA BLIP image-text retrieval
Closest papers by meaning
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.93)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.90)
- DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering (CVPR 2025 · similarity 0.89)
- Zero-shot 3D Question Answering via Voxel-based Dynamic Token Compression (CVPR 2025 · similarity 0.89)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.89)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.