Research map / Vision-language and multimodal / Multimodal LLMs
Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Mutimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Spatial457, a synthetic, pose-balanced VQA benchmark with seven question types across five difficulty levels for 6D spatial reasoning in large multimodal models, plus an RPDR metric and bias analysis.
Problem
Existing VQA benchmarks cover 2D relations or limited 2.5D/3D cues, and real-image datasets have skewed object orientations, so LMMs' 6D (3D position + 3D orientation) spatial reasoning is not comprehensively evaluated.
Approach
A scalable, orientation-balanced synthetic dataset rendered with realistic HDRI backgrounds and textured meshes, organized as a cascading roadmap that adds multi-object, 2D location, 3D location (occlusion), and 3D orientation factors, with new 3D-relate and collision questions and a relative performance dropping rate metric.
Key results
- GPT-4o reaches 74.46% on L1 single-object and 37.01% on L5 6D spatial questions, versus 79.94% human on 6D spatial questions.
- PO3D-VQA neural symbolic model reaches 71.06% on 6D spatial questions, the highest among evaluated systems, but is not a fair LMM comparison.
- RPDR analysis shows GPT-4o and GeminiPro-1.5 weakest on 3D orientation (77.46% and 83.28%), while Claude 3.5 Sonnet and Qwen2-VL-7B are weakest on 3D location (72.05% and 74.40%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add multi-object reasoning to single-object questions | LMMs evaluated on Spatial457 · L1 to L2 | accuracy: 74.46 -> 62.88 | Hurt |
| Add 2D spatial relations to multi-object questions | LMMs evaluated on Spatial457 · L2 to L3 | accuracy: 62.88 -> 56.14 | Hurt |
| Add 3D location reasoning through occlusion questions | LMMs evaluated on Spatial457 · L3 to L4 Occlusion | accuracy: 56.14 -> 48.40 | Hurt |
| Add 3D orientation reasoning | LMMs evaluated on Spatial457 · L3 to L4 3D Pose | accuracy: 56.14 -> 42.41 | Hurt |
| Combine 3D location and orientation in 6D spatial questions | LMMs evaluated on Spatial457 · L4 3D Pose to L5 6D Spatial | accuracy: 42.41 -> 37.01 | Hurt |
| Add collision prediction to spatial reasoning | LMMs evaluated on Spatial457 · L4 Occlusion to L5 Collision | accuracy: 48.40 -> 38.41 | Hurt |
| Use 3D pose reasoning instead of 2D spatial reasoning | LMMs evaluated on Spatial457 · L3 2D Spatial to L4 3D Pose | accuracy: 54.49 -> 43.67 | Hurt |
| Add collision prediction to spatial reasoning | LMMs evaluated on Spatial457 · L4 Occlusion to L5 Collision | accuracy: 47.65 -> 41.19 | Hurt |
| Combine 3D location and orientation in 6D spatial questions | LMMs evaluated on Spatial457 · L4 3D Pose to L5 6D Spatial | accuracy: 43.67 -> 39.36 | Hurt |
| Evaluate 2D spatial relations with InternVL2-8B instead of API models | InternVL2-8B versus API models · L3 2D Spatial | accuracy: 57.40 vs 56.14 | Helped |
| Use PO3D-VQA rather than GPT-4o | Spatial457 evaluation · L5 6D Spatial | accuracy: 37.01 -> 71.06 | Helped |
| Use human reasoning rather than GPT-4o | Spatial457 evaluation · L5 6D Spatial | accuracy: 37.01 -> 79.94 | Helped |
Surprising findings
- Claude 3.5 Sonnet and Qwen2-VL show weakest RPDR on 3D location while GPT-4o and Gemini are weakest on 3D orientation, so bottlenecks differ across models.
- Models show strong prediction biases (e.g., yellow and front overpredicted) even though ground-truth labels are balanced.
Limitations
- Real-image extension is limited to the L4-Pose-Real pose questions on SUN-RGBD due to missing annotations for other attributes.
- Comparison with PO3D-VQA is not fully fair since its modules are trained independently on sub-tasks.
- Evaluated only a limited set of LMMs, with open-source models mostly 7-8B.
Experimental setup
Datasets: SUN-RGBD
Benchmarks: Spatial457 SUN-RGBD-based L4-Pose-Real
Metrics: accuracy relative performance dropping rate (RPDR) coefficient of variation (CV)
Baselines: GPT-4o Gemini-1.5-Pro Claude-3.5-Sonnet Qwen2-VL-7B-Instruct InternVL2-8B LLaVA-v1.5-7B LLaVA-NeXT-vicuna-7B LLaVA-NeXT-llama3-8B PO3D-VQA
Builds on: CLEVR Super-CLEVR-3D PO3D-VQA VLMEvalKit
Closest papers by meaning
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.91)
- SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models (CVPR 2025 · similarity 0.91)
- SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning (NeurIPS 2025 · similarity 0.91)
- Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests (NeurIPS 2026 · similarity 0.90)
- SpaceVista: All-Scale Visual Spatial Reasoning from mm to km (ICML 2026 · similarity 0.90)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.