Research map / Vision-language and multimodal / Multimodal LLMs
OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OST-Bench, a 10k-question online spatio-temporal benchmark built from 1.4k indoor scene exploration trajectories, and finds leading MLLMs lag far behind humans, especially on agent-state and spatial-relationship reasoning.
Problem
Existing spatial and video benchmarks evaluate MLLMs offline over fixed inputs, which does not capture the online, memory-dependent spatio-temporal reasoning an embodied agent needs while exploring a scene.
Approach
A multi-round dialogue benchmark where frames stream in incrementally and questions are asked from the agent's current situation about its state, visible information, and spatial relations to objects, with rule-based generation from 9-DoF box and MMScan annotations over synthesized or recorded exploration trajectories.
Key results
- Best proprietary model GPT-4.1 averages 53.4 overall versus 83.5 for human-level on OST-Bench.
- Open-source InternVL-2.5-78B reaches 51.1 overall; QwenVL-2.5-7B reaches 41.2.
- Fine-tuning on 50k generated samples raises QwenVL-2.5-7B from 41.2 to 54.0 and InternVL-2.5-38B from 50.8 to 60.2 overall, still below human level.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on 50k OST-Bench samples for one epoch (ablation) | QwenVL2.5-7B · OST-Bench overall | accuracy: 41.2 -> 54.0 | Helped |
| fine-tune on 50k OST-Bench samples for one epoch (ablation) | InternVL2.5-8B · OST-Bench overall | accuracy: 44.6 -> 57.4 | Helped |
| fine-tune on 50k OST-Bench samples for one epoch (ablation) | InternVL2.5-38B · OST-Bench overall | accuracy: 50.8 -> 60.2 | Helped |
| enable Thinking mode | Gemini-2.0-Flash · OST-Bench overall | accuracy: 49.5 -> 54.2 | Helped |
| scale InternVL-2.5 from 8B to 38B | InternVL-2.5 · OST-Bench overall | accuracy: 44.6 -> 50.8 | Helped |
| use spatial grounding architecture and spatial pretraining | QwenVL2.5-3B / Spatial-MLLM · OST-Bench overall | accuracy: 34.8 -> 26.8 | Hurt |
| use VLM-3R spatial grounding model | LLaVA-Video-7B · OST-Bench overall | accuracy: 39.3 -> 42.9 | Helped |
| use LLaVA-3D spatial grounding model | LLaVA-Video-7B · OST-Bench overall | accuracy: 39.3 -> 30.1 | Hurt |
| require multi-step instead of single-step spatial connections | Gemini-2.0-Flash (Thinking), GPT-4o, and Claude-3.5-Sonnet · Cross-view subset | accuracy | Hurt |
| provide sequence-based memory instead of directly provided keyframes | Gemini-2.0-Flash (Thinking), GPT-4o, and Claude-3.5-Sonnet · Cross-view subset | accuracy | Hurt |
| combine multi-step spatial connections with sequence-based context | Gemini-2.0-Flash (Thinking), GPT-4o, and Claude-3.5-Sonnet · Cross-view subset | accuracy: around 10% | Hurt |
| increase the exploration horizon and accumulated memory | MLLMs evaluated on OST-Bench · OST-Bench online exploration | accuracy | Hurt |
Surprising findings
- Only VLM-3R improves over its base model among spatially specialized models; Spatial-MLLM and LLaVA-3D drop substantially, showing poor generalization.
- Fine-tuning gains on estimation and judgement tasks appear to reflect dataset shortcuts or memorization rather than genuine reasoning.
Limitations
- Assumes a static environment; object states and positions do not change.
- Focuses only on perception and reasoning, excluding interaction and manipulation.
- Matterport3D trajectories are synthesized rather than recorded videos.
Experimental setup
Datasets: ScanNet Matterport3D ARKitScenes EmbodiedScan MMScan
Benchmarks: OST-Bench
Metrics: accuracy Mean Relative Accuracy (MRA)
Baselines: Claude-3.5-Sonnet Gemini-2.0-Flash GPT-4o GPT-4.1 InternVL-2.5 QwenVL-2.5 LLaVA-Video LLaVA-Onevision Spatial-MLLM VLM-3R LLaVA-3D
Builds on: VSI-Bench EmbodiedScan MMScan MST (Prim's algorithm)
Closest papers by meaning
- Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model (CVPR 2025 · similarity 0.92)
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.92)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.91)
- InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.91)
- GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.