Research map / Vision-language and multimodal / Multimodal LLMs
Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Spatial Intelligence Grid (SIG), a grid-based schema for autonomous-driving visual-spatial intelligence, with graph-based metrics (MLSM, SRGS, SRD) and SIGBench (1,423 frames with SIG and gaze labels), showing SIG-based few-shot ICL beats VQA-style ICL on MLLMs.
Problem
Visual-spatial intelligence in foundation models is usually evaluated with text-based VQA, which conflates grounding errors with spatial reasoning and invites language shortcuts; no structured geometric representation or metric exists for verifying it, especially in autonomous driving.
Approach
Encode driving scenes as a 10x10 grid (SIG) of object positions plus directed spatial relation graphs and paragraphs, use it as a complementary representation for in-context examples, and score model outputs with graph-based metrics (multi-level spatial matching, graph edit distance similarity, semantic relational distance) that separate spatial skill from language priors; add human gaze attention
Key results
- On SIGBench, Gemini-2.5-Pro achieves best MLSM F1 (0.507) and directional SRD accuracy (0.254) among zero-shot models, while GPT-4o leads SRGS (S=0.337).
- 3-shot ICL-SIG with GPT-4o on SIGBench-tiny raises MLSM F1 from 0.462 to 0.479 and directional SRD accuracy from 0.186 to 0.220, outperforming ICL-MC.
- With Gemini-2.5-Pro, ICL-SIG improves MLSM F1 from 0.496 to 0.565 and SRGS S from 0.224 to 0.305, while ICL-MC drops SRGS S to 0.185 (below zero-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SIG rather than multiple-choice VQA examples for 3-shot ICL | GPT-4o · SIGBench-tiny | F1: 0.468 -> 0.479 | Helped |
| Use SIG rather than multiple-choice VQA examples for 3-shot ICL | GPT-4o · SIGBench-tiny | SRGS S: 0.324 -> 0.337 | Helped |
| Use SIG rather than multiple-choice VQA examples for 3-shot ICL | Gemini-2.5-Pro · SIGBench-tiny | SRGS S: 0.185 -> 0.305 | Helped |
| Use SIG rather than multiple-choice VQA examples for 3-shot ICL | Gemini-2.5-Pro · SIGBench-tiny | Directional MAE: 1.174 -> 1.126 | Helped |
| Use SIG examples for 3-shot ICL rather than zero-shot inference | GPT-4o · SIG-COCO | F1: 0.771 -> 0.849 | Helped |
| Use SIG examples for 3-shot ICL rather than zero-shot inference | GPT-4o · SIG-ARKitScenes | F1: 0.727 -> 0.823 | Helped |
Surprising findings
- ICL with multiple-choice VQA examples lowered SRGS and directional SRD performance below zero-shot for both GPT-4o and Gemini-2.5-Pro.
- Some open-source models (InternVL2.5-26B, Qwen-VL-2.5-7B) top gaze prediction merely by averaging previous attention maps without the smoothing other models apply.
Limitations
- SIGBench is single-frame and does not assess tracking or temporal object relations.
- SIG-based fine-tuning and RLHF are unexplored; only few-shot ICL was tested.
- Several open-source models (LLaMA-3.2-11B-Vision, DeepSeek-VL) could not produce valid SIG JSON and are missing from results.
Experimental setup
Datasets: SIGBench (from FHWA driving dataset) MS COCO ARKitScenes
Benchmarks: SIGBench SIGBench-tiny SIG-COCO SIG-ARKitScenes
Metrics: precision recall F1 AssA similarity (S) weighted similarity (WS) MAE MSE accuracy PCC
Baselines: Claude-3.5-Haiku Claude-3.7-Sonnet Gemini-1.5-Pro Gemini-2.0-Flash Gemini-2.5-Pro GPT-4o-mini GPT-4o InternVL3-9B InternVL3-14B InternVL2.5-26B Qwen-VL-2.5-7B Qwen-VL-2.5-32B
Builds on: in-context learning graph edit distance MOTA HOTA
Closest papers by meaning
- Scaling Spatial Intelligence with Multimodal Foundation Models (CVPR 2026 · similarity 0.89)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.89)
- Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests (NeurIPS 2026 · similarity 0.89)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.89)
- Thinking in Structures: Evaluating Spatial Intelligence in Constraint-Governed Spaces (ICML 2026 · similarity 0.89)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.