Research map / Vision-language and multimodal / Visual question answering
Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BridgeQA, a 3D VQA model that selects question-relevant 2D views via BLIP retrieval and fuses 2D and 3D branches with a Twin-Transformer initialized from BLIP.
Problem
3D visual question answering suffers from scarce annotated 3D scenes (about 800 in ScanQA and SQA), and existing 2D-augmented methods use top-down views or global features that lose fine-grained vision-language correspondence.
Approach
Select question-conditional 2D views with a BLIP image-text retriever, and fuse them with 3D object tokens in a two-branch BLIP-initialized transformer whose layers exchange intermediate tokens via concatenation/cross-attention (Twin-Transformer), so both branches mutually augment each other.
Key results
- Achieves 31.29 EM@1 on ScanQA test w/ objects and 30.82 on test w/o objects, outperforming prior second-best (3D-VisTA 27.0 and 23.0).
- Achieves 52.91 EM@1 on SQA test, versus 48.5 for 3D-VisTA (about 4.4% margin).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use question-to-declaration transformation for 2D view selection (ablation) | BridgeQA · ScanQA val | EM@1: 26.90 -> 26.98 | Helped |
| Include the answer when transforming questions into view-selection declarations (ablation) | BridgeQA · ScanQA val | EM@1: 25.60 vs 26.98 | Hurt |
| Add 2D visual input to the 3D-only model (ablation) | BridgeQA · ScanQA val | EM@1: 13.99 -> 26.98 | Helped |
| Add 3D visual input to the 2D-only model (ablation) | BridgeQA · ScanQA val | EM@1: 25.72 -> 26.98 | Helped |
| Use Twin-Transformer fusion instead of single-stream concatenation (ablation) | BridgeQA · ScanQA val | EM@1: 25.90 -> 26.98 | Helped |
| Add intermediate feature cross-attention to simple two-branch fusion (ablation) | BridgeQA · ScanQA val | EM@1: 26.45 -> 26.98 | Helped |
| Use a pretrained language decoder instead of an MLP classification head (ablation) | BridgeQA · ScanQA val | EM@1: 21.37 -> 26.98 | Helped |
| Share the answer decoder between 2D and 3D branches (ablation) | BridgeQA · ScanQA val | EM@1: 26.49 -> 26.98 | Helped |
| Use BridgeQA instead of 3D-VisTA | 3D VQA · ScanQA test w/ objects | EM@1: 27.0 -> 31.29 | Helped |
| Use BridgeQA instead of 3D-VisTA | 3D VQA · ScanQA test w/o objects | EM@1: 23.0 -> 30.82 | Helped |
| Use BridgeQA instead of 3D-VisTA | 3D VQA · ScanQA val | EM@1: 24.03 -> 26.98 | Helped |
| Use BridgeQA instead of 3D-VisTA | 3D VQA · SQA test | EM@1: 48.5 -> 52.91 | Helped |
Surprising findings
- A 2D-only variant (25.72 EM@1) performs far better than a 3D-only variant (13.99) on ScanQA validation, suggesting 3D tokens alone are weak under limited data.
Limitations
- Relies on a 3D object detector (VoteNet) and on 2D views from video frames, so performance depends on view retrieval quality.
- Question-to-declaration conversion introduces an extra GPT-4 step during training.
Experimental setup
Datasets: ScanQA SQA ScanNet
Benchmarks: ScanQA SQA
Metrics: EM@1 BLEU-1 BLEU-4 ROUGE METEOR CIDEr
Baselines: ScanQA SQA FE-3DGQA CLIP-Guided Multi-CLIP 3DVLP 3D-VisTA
Builds on: BLIP VoteNet ScanQA CLIP-Guided 3D-VQA
Closest papers by meaning
- 3D Question Answering via only 2D Vision-Language Models (ICML 2025 · similarity 0.93)
- Towards Robust Visual Question Answering via Prompt-Driven Geometric Harmonization (AAAI 2025 · similarity 0.91)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
- DSPNet: Dual-vision Scene Perception for Robust 3D Question Answering (CVPR 2025 · similarity 0.90)
- Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning (CVPR 2026 · similarity 0.90)
- Language-Image Models with 3D Understanding (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.