Research map / Vision-language and multimodal / Visual question answering
Hypo3D: Exploring Hypothetical Reasoning in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Hypo3D, a 3D VQA benchmark where models must imagine a described scene change before answering questions, and shows that ten foundation models lag far behind humans, with models also hallucinating answer changes when context changes are irrelevant.
Problem
Existing 3D reasoning benchmarks assume real-time access to up-to-date scene data, which is impractical; models need to reason about hypothetically changed 3D scenes without re-observing them.
Approach
A hypothetical 3D reasoning task in which a textual context change is applied to a scene the model cannot re-observe, combined with an anchor-based world frame that fixes directional terms to a global reference rather than an observer or object-centric frame.
Key results
- GPT-4o with semantic top-view map achieves the best overall EM of 45.50%, versus 91.00% for human evaluators.
- Movement changes are hardest for models; models score about 9.8 EM points lower on movement than on addition changes (best-model overall numbers).
- Adding irrelevant context changes lowers accuracy for all 2D and 3D VLMs, e.g., GPT-4o drops from 57.00 to 52.50 EM on questions unaffected by the change.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a semantic rather than non-semantic top-view map (ablation) | GPT-4o API · Hypo3D overall | EM: 33.58 -> 45.50 | Helped |
| Use a semantic rather than non-semantic top-view map (ablation) | GPT-4o API · Hypo3D overall | PM: 36.75 -> 48.82 | Helped |
| Increase model size from 7B to 72B (ablation) | Qwen2-VL with semantic top-view maps · Hypo3D overall | EM: 34.40 -> 44.25 | Helped |
| Use the anchor-based world frame instead of no frame (ablation) | Qwen2-VL 72B · Directional questions | EM: 20.54 -> 22.03 | Helped |
| Replace the anchor frame with a camera-view frame (ablation) | Qwen2-VL 72B · Directional questions | EM: 20.54 -> 19.97 | Hurt |
| Reason with the context change rather than without it (ablation) | Qwen2-VL 72B · Changed-scene questions | EM: 37.00 -> 31.50 (-5.50) | Hurt |
| Reason with the context change rather than without it (ablation) | GPT-4o API · Changed-scene questions | EM: 38.00 -> 33.00 (-5.00) | Hurt |
| Reason with the context change rather than without it (ablation) | LLaMA-3.2 3B · Changed-scene questions | EM: 19.00 -> 20.50 (+1.50) | Helped |
| Include an irrelevant context change (ablation) | Qwen2-VL 72B · Irrelevant-change questions | EM: 56.50 -> 51.50 (-5.00) | Hurt |
| Include an irrelevant context change (ablation) | GPT-4o API · Irrelevant-change questions | EM: 57.00 -> 52.50 (-4.50) | Hurt |
| Include an irrelevant context change (ablation) | LLaVA-3D 7B · Irrelevant-change questions | PM: 40.17 -> 40.17 (0.00) | No clear effect |
Surprising findings
- Closed-source models (GPT-4o, Claude 3.5 Sonnet) do not consistently beat open-source models, and perform worse than LLaVA-OV 72B on non-semantic top-view maps.
- Models often change answers when the context change is irrelevant to the question, indicating hallucination of scene modifications.
Limitations
- Human evaluation was on a subset of 50 scenes and 250 context changes, and humans were not evaluated on the full benchmark.
- Models are evaluated only zero-shot with fixed prompts and no training; the study does not test whether fine-tuning could improve hypothetical reasoning.
- Benchmark relies on GPT-4o for raw question and change generation and on GPT-4-Turbo for grammar refinement.
Experimental setup
Datasets: ScanNet 3RScan ScanRefer SceneVerse
Benchmarks: Hypo3D
Metrics: Exact Match Partial Match GPT score SBERT similarity
Baselines: Llama-3.2 3B GPT-4o Qwen2-VL 7B Qwen2-VL 72B LLaVA-OV 7B LLaVA-OV 72B Claude 3.5 Sonnet LEO LLaVA-3D Human
Builds on: ScanQA SQA3D ScanRefer LEO LLaVA-3D
Closest papers by meaning
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.90)
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.89)
- Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis (CVPR 2025 · similarity 0.89)
- 3D Question Answering via only 2D Vision-Language Models (ICML 2025 · similarity 0.89)
- Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answering (ICLR 2025 · similarity 0.88)
- SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.