Research map / Large language models / Agents, code and math
Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STARK, a hierarchical spatiotemporal reasoning benchmark (26 tasks, 14,552 instances) evaluating 8 LLMs and 3 LRMs in direct-answer and code-interpreter modes against first-principle baselines.
Problem
LLMs and LRMs' ability to reason jointly over space, time, and world knowledge for cyber-physical system tasks (state estimation, reasoning over states, knowledge-aware reasoning) is underexplored, and existing benchmarks are fragmented or multiple-choice.
Approach
A three-tier, open-ended benchmark that covers localization, tracking, field-variable prediction, DE-9IM spatial and Allen-interval temporal relations, and world-knowledge tasks, scored by RMSE/RMSPE in both direct-answer and Python code-interpreter modes.
Key results
- o3 achieves the lowest error on most localization and tracking tasks on STARK-S, e.g. Loc-range RMSE 0.036 versus 1.79 for gpt-4o.
- Code interpreter cuts localization error by nearly 80% for gpt-4o, gpt-4.1, and gpt-4o-mini, but generally degrades tracking performance.
- LRMs often match or beat first-principle baselines on range-bearing tracking, while LLMs struggle on geometric localization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Python code interpreter for localization instead of direct answering | LLMs including GPT-4o, GPT-4.1, and GPT-4o-mini · STARK-S localization tasks | RMSE: nearly 80% | Helped |
| Use Python code interpreter for tracking instead of direct answering | Evaluated models · STARK-S tracking tasks | RMSE | Hurt |
| Use Python code interpreter instead of direct answering | o3-mini · STARK-S tasks | RMSE: up to +86% | Mixed |
| Use a larger reasoning model (o3 rather than smaller LRMs) | LRMs · STARK tasks | RMSE | Helped |
| Use LRMs rather than LLMs | Models evaluated on localization and tracking · STARK-S localization and tracking tasks | RMSE: 3× to 10× lower error | Helped |
| Use LRMs rather than first-principle methods | Localization range task · STARK-S Loc-range | RMSE | Helped |
| Use LLMs rather than LRMs for world-knowledge reasoning | Models evaluated on Tier 3 tasks · STARK-S world-knowledge tasks | RMSE | Mixed |
Surprising findings
- Code interpreter can increase error substantially for tracking tasks, with o3-mini error rising up to +86% in some cases.
- Smaller LRM o3-mini lags behind some LLMs on world-knowledge tasks such as intent prediction and POI prediction.
Limitations
- Model sizes for o3 and other closed models are undisclosed, so scale claims are speculative.
- Many tasks rely on simulated sensor data and synthetic trajectories rather than real-world measurements.
- Code interpreter outputs are unstable and need guardrails to avoid severe errors.
Experimental setup
Datasets: PurpleAir air quality CalTrans PeMS traffic OpenStreetMap ArcGIS
Benchmarks: STARK-S STARK-L
Metrics: RMSE RMSPE relative change in RMSE with code interpreter
Baselines: multilateration triangulation Extended Kalman Filter finite state machine random guess
Builds on: DE-9IM Allen's interval algebra SCAAT tracking Extended Kalman Filter
Closest papers by meaning
- Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models (ICLR 2026 · similarity 0.86)
- From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs (CVPR 2026 · similarity 0.86)
- EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer (AAAI 2026 · similarity 0.86)
- SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.86)
- GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.86)
- ChainSpace: A Chained-Reasoning Paradigm for Spatial Intelligence (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.