Research map / Large language models / Agents, code and math
USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning Capabilities of LLMs as Urban Agents
Siqi Lai, Yansong Ning, Zirui Yuan, Zhixi Chen, Hao Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use reasoning-post-trained models (R1, QwQ, GLM-Z1) instead of their non-reasoning base models for spatiotemporal understanding | LLM urban agents · USTBench spatiotemporal understanding | accuracy: gains of 7-20% | Improved |
| use reasoning model DeepSeek-R1 instead of non-reasoning Llama3.3-70B on temporal trend understanding | DeepSeek-R1 · USTBench Trend understanding | accuracy: 0.3203 -> 0.2502 | Hurt |
| use reasoning variants (QwQ, DeepSeek-R1) instead of non-reasoning bases on congestion prediction | QwQ-32B / DeepSeek-R1 · USTBench congestion forecasting | accuracy: Qwen2.5-32B 0.6900 -> QwQ-32B 0.6290; Llama3.3-70B 0.7493 -> DeepSeek-R1 0.6020 | Hurt |
| use reasoning variants instead of non-reasoning bases on planning overall | DeepSeek-R1 vs Llama3.3-70B · USTBench planning | accuracy: 0.4503 -> 0.3968 | Hurt |
| post-train Qwen2.5-7B on synthetic spatiotemporal understanding data (Qwen2.5-7B-ST) | Qwen2.5-7B · USTBench downstream reasoning | accuracy | Improved |
| add reflection mechanism to agent with strong reflection ability (ablation) | DeepSeek-R1 agent · urban downstream tasks | task performance | Improved |
| add reflection mechanism to agent with moderate reflection ability (ablation) | Qwen2.5-32B agent · urban downstream tasks | task performance | No clear effect |
| add reflection mechanism to agent with weak reflection ability (ablation) | Qwen2.5-7B agent · urban downstream tasks | task performance | Hurt |
| distill DeepSeek-R1 into Qwen2.5-7B (R1-Distill-Qwen-7B) instead of Qwen2.5-7B | Qwen2.5-7B · USTBench spatiotemporal understanding | accuracy: 0.4552 -> 0.2936 | Hurt |
| scale model size from Qwen2.5-7B to Qwen2.5-32B | Qwen2.5 · USTBench spatiotemporal understanding | accuracy: 0.4552 -> 0.6224 | Improved |
| use LLM agents instead of classic domain baselines on downstream urban tasks | LLM urban agents · Congestion prediction | accuracy (%): 17.18% -> 75.73% (o4-mini) | Improved |
Closest papers by meaning
- Benchmarking Spatiotemporal Reasoning in LLMs and Reasoning Models: Capabilities and Challenges (NeurIPS 2025 · similarity 0.88)
- Get Rid of Isolation: A Continuous Multi-task Spatio-Temporal Learning Framework (NeurIPS 2024 · similarity 0.87)
- Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark (AAAI 2024 · similarity 0.87)
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.87)
- MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models (ICML 2025 · similarity 0.87)
- ACPBench: Reasoning About Action, Change, and Planning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
