Research map / Point clouds / Point cloud classification
DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSPNet, a dual-vision 3D QA model that fuses back-projected multi-view image features with point cloud features using text-guided view weighting, adaptive channel/point gating, and multimodal context-guided reasoning.
Problem
3D question answering relies mainly on point clouds and loses local texture detail from multi-view images; naive back-projection fusion suffers from camera-pose noise, occlusion, and fixed view weighting, causing feature degradation.
Approach
Text-guided attention weights over back-projected multi-view features (TGMF), SENet-style point-wise and channel-wise gating to fuse image and point features (ADVP), and a cross-attention plus transformer reasoning stack over FPS-sampled visual tokens and text (MCGR).
Key results
- On SQA3D, DSPNet reaches 50.4% average accuracy versus 49.2% for 3DGraphQA and 48.5% for pre-trained 3D-VisTA.
- On ScanQA, DSPNet reaches EM@1 26.5 / 23.8 (w/ / w/o object) and CIDEr 78.1 / 69.6, the best among compared methods.
- Ablation shows removing multi-view images drops SQA3D from 50.36 to 49.05 and ScanQA EM@1 from 23.47 to 22.26.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TGMF to the baseline (ablation) | DSPNet · ScanQA validation | EM@1: 22.35 -> 22.69 | Helped |
| Add TGMF to the baseline (ablation) | DSPNet · SQA3D test | accuracy: 49.33 -> 49.58 | Helped |
| Add ADVP after TGMF (ablation) | DSPNet · ScanQA validation | EM@1: 22.69 -> 22.80 | Helped |
| Add ADVP after TGMF (ablation) | DSPNet · SQA3D test | accuracy: 49.58 -> 49.87 | Helped |
| Add MCGR after TGMF (ablation) | DSPNet · ScanQA validation | EM@1: 22.69 -> 23.23 | Helped |
| Add MCGR after TGMF (ablation) | DSPNet · SQA3D test | accuracy: 49.58 -> 49.77 | Helped |
| Use multi-view images in addition to point clouds (ablation) | DSPNet · ScanQA validation | EM@1: 22.26 -> 23.47 | Helped |
| Use multi-view images in addition to point clouds (ablation) | DSPNet · SQA3D test | accuracy: 49.05 -> 50.36 | Helped |
| Increase multi-view images from 10 to 20 (ablation) | DSPNet · ScanQA validation | EM@1: 22.87 -> 23.47 | Helped |
| Increase multi-view images from 10 to 20 (ablation) | DSPNet · SQA3D test | accuracy: 49.73 -> 50.36 | Helped |
| Increase MCGR depth from 4 to 6 (ablation) | DSPNet · ScanQA validation | EM@1: 23.47 -> 22.48 | Hurt |
| Increase MCGR depth from 4 to 6 (ablation) | DSPNet · SQA3D test | accuracy: 50.36 -> 49.30 | Hurt |
Surprising findings
- Deeper MCGR (6 layers) hurts performance, which the authors attribute to overfitting on limited 3D QA data.
Limitations
- Gains over prior methods are modest (e.g., EM@1 +0.9 over the baseline on ScanQA in ablations).
- Performance on Is/Can/Which questions on SQA3D is lower than some baselines.
Experimental setup
Datasets: ScanQA SQA3D ScanNet
Benchmarks: SQA3D ScanQA
Metrics: answer accuracy by question type EM@1 EM@10 BLEU-4 ROUGE METEOR CIDEr
Baselines: ClipBERT MCAN ScanQA SQA3D Multi-CLIP 3D-VisTA 3DGraphQA
Builds on: ScanQA SQA3D 3DMV VoteNet SENet
Closest papers by meaning
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.90)
- 3D Question Answering via only 2D Vision-Language Models (ICML 2025 · similarity 0.89)
- MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations (NeurIPS 2024 · similarity 0.88)
- Hypo3D: Exploring Hypothetical Reasoning in 3D (ICML 2025 · similarity 0.87)
- Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs (NeurIPS 2025 · similarity 0.87)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.