Research map / Vision-language and multimodal / Multimodal LLMs

SAW-Bench: Learning Situated Awareness in the Real World

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Spatial reasoning in VLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SAW-Bench, a 2,071-QA benchmark over 786 self-recorded Ray-Ban Meta egocentric videos testing observer-centric situated awareness in multimodal foundation models, finding a large human-model gap.

Problem

Existing spatial benchmarks for multimodal foundation models focus on environment-centric relations and static third-person views, leaving observer-centric situated awareness (position, orientation, and motion relative to the observer) largely unevaluated.

Approach

A video benchmark built from self-recorded egocentric smart-glasses videos following predefined camera trajectories, with six observer-centric tasks (self-localization, relative direction, route shape, reverse route plan, spatial memory, spatial affordance) and human-annotated multiple-choice QA.

Key results

• Gemini 3 Flash achieves the best model accuracy of 53.89% on SAW-Bench versus 91.55% human performance (37.66-point gap).
• Proprietary models generally outperform open-source ones; Qwen3-VL 235B-A22B reaches 41.40% overall as the best open-source model.
• Blind LLM (GPT-5.2) and Socratic captioning baseline both reach 31.34%, only slightly above chance (27.49%).

Surprising findings

• Top models misclassify a straight-path video with head rotations as zigzag in 60% of cases, conflating camera rotation with translation.
• Outdoor scenes do not yield consistently lower performance than indoor scenes; environment openness is an insufficient proxy for difficulty.

Limitations

• Small scene coverage (10 outdoor and 5 indoor scenes) and limited video count may restrict generality.
• Evaluation is zero-shot multiple-choice only; no training or open-ended answers are studied.
• Human baseline uses only two graduate students.

Experimental setup

Datasets: SAW-Bench (786 self-recorded egocentric videos, 2,071 QA pairs)

Benchmarks: SAW-Bench

Metrics: accuracy

Baselines: Blind LLM (GPT-5.2) Socratic Model (GPT-5.2) Chance level (random) Chance level (frequent) Human level

Builds on: TOMATO HourVideo Socratic Models

Closest papers by meaning

• SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing (NeurIPS 2025 · similarity 0.88)
• Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.88)
• From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs (CVPR 2026 · similarity 0.88)
• Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.88)
• EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy (CVPR 2026 · similarity 0.88)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.88)