Research map / Vision-language and multimodal / Multimodal LLMs
Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces COMFORT, a synthetic 3D evaluation protocol with consistency and robustness metrics to test how nine VLMs resolve frame-of-reference ambiguities in English and 109 languages.
Problem
Spatial expressions in situated communication are ambiguous because speakers may adopt different frames of reference (FoR) and languages follow different coordinate conventions, yet VLM spatial evaluations ignore this ambiguity, robustness, and consistency.
Approach
A rotating-referent synthetic scene protocol (COMFORT) with region-parsing error against hemisphere and cosine acceptance references, plus robustness (std, noise) and consistency (symmetry, opposite) metrics, used to infer which FoR and coordinate transformation VLMs prefer, in English and across 109 languages.
Key results
- Most VLMs (except BLIP series) prefer the egocentric relative FoR with reflected transformation, matching English conventions.
- All models, including GPT-4o and XComposer2, perform near chance (~50% accuracy) when prompted to use intrinsic or addressee-centered FoRs.
- GPT-4o's opposite consistency is not significantly better than 30 random trials.
- GPT-4o's English-based preferences extend across 109 languages, with near-random preference for Konkani, Kurdish, and Amharic.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reflected coordinate transformation rather than same-direction, rotated, or translated mappings | Most evaluated VLMs on COMFORT-BALL · COMFORT-BALL, English | cosine region parsing error: Aggregated: Ref. 20.7; Tran. 34.5; Rot. 49.0 (LLaVA-1.5-7B) | Helped |
| Leave the frame of reference unspecified instead of prompting for intrinsic or addressee-centered relative FoR | VLMs on COMFORT-CAR · COMFORT-CAR, English | accuracy: Compared to nop: accuracy decreases with explicit FoR prompts; GPT-4o aggregated accuracy 60.3 vs. 78.3 egocentric | Helped |
| Specify alternative intrinsic or addressee-centered perspective | VLMs on COMFORT-CAR · COMFORT-CAR, English | cosine region parsing error: GPT-4o aggregated: 39.2 nop; 50.9 intrinsic; 51.3 addressee | Mixed |
| Use spatial-task supervised fine-tuning | InternLM-XComposer2 and MiniCPM-V · COMFORT-BALL | prediction noise: 13.7 (InternLM-XComposer2); 29.8 (MiniCPM-V) | Helped |
| Rely on object grounding without spatial-relation training | GLaMM · COMFORT-BALL and COMFORT-CAR | accuracy: 47.2 (COMFORT-BALL); 47.2 (COMFORT-CAR) | Hurt |
| Evaluate across different spatial relations rather than treating directions as uniform | VLMs on COMFORT-CAR · COMFORT-CAR, English | cosine region parsing error: GPT-4o egocentric: back 38.3; left 34.7 | Mixed |
| Use translated multilingual prompts instead of English prompts | GPT-4o · 109 languages, 170 regions | preferred coordinate transformation | Hurt |
| Use language-specific and cultural FoR conventions rather than English-dominant conventions | GPT-4o multilingual evaluation · 109 languages | convention adherence | Hurt |
| Use explicit camera perspective with reflected transformation | GPT-4o · COMFORT-CAR | accuracy: 78.3 | Helped |
Surprising findings
- GPT-4o's opposite consistency is not significantly better than random baselines.
- Models perform better on lateral (left/right) than sagittal (front/behind) relations.
Limitations
- Acceptance regions (cosine and hemisphere) are analytical and may not capture human cognitive biases or cultural variation.
- Only four basic relations (left, right, front, behind) and minimal occlusion are studied.
- Multilingual prompts were machine-translated via Google Cloud Translate and only partially verified by back-translation.
Experimental setup
Datasets: COMFORT-BALL (720 English test cases) COMFORT-CAR (57.6k English test cases)
Benchmarks: COMFORT-BALL COMFORT-CAR
Metrics: accuracy region parsing error (cosine, hemisphere) object F1 prediction noise standard deviation symmetry consistency opposite consistency
Baselines: InstructBLIP mBLIP GLaMM LLaVA-1.5 InternLM-XComposer2 MiniCPM-V
Builds on: Blender object hallucination probing (POPE-style)
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.93)
- SpatiaLab: Can Vision–Language Models Perform Spatial Reasoning in the Wild? (ICLR 2026 · similarity 0.92)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.90)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.