Research map / Vision-language and multimodal / Multimodal LLMs
STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces STSBench, an automatic scenario-mining and human-verification framework, instantiated as STSnu, a 971-question multiple-choice benchmark on NuScenes for spatio-temporal reasoning of VLMs and driving expert models.
Problem
Existing driving VLM benchmarks focus on single images or monocular video and ego actions, and are tied to specific datasets, so they do not test holistic spatio-temporal reasoning about multi-view video or LiDAR scenes, including agent-to-agent interactions.
Approach
Mine predefined traffic scenarios automatically from any dataset's ground-truth annotations (3D boxes, tracks, ego-motion, HD maps) using a scenario catalog with hard negatives, verify them with a lightweight visual tool, and auto-generate multiple-choice questions.
Key results
- GPT-4o reaches the highest average accuracy of 57.08% on STSnu, while the best driving expert (DriveMM) reaches 39.51%.
- Driving expert VLMs struggle on ego and agent-to-agent scenarios; DriveMM scores 43.49% on ego scenarios versus 63.63% for GPT-4o.
- Human verifiers agreed on 85.6% of positive samples and disagreed on 20.8% of negatives, with about 14.5 seconds per sample and inspector.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ground-truth trajectory information instead of visual inputs | GPT-4o versus DriveMM · STSnu Ego | accuracy: 43.49% vs. 63.63% | Helped |
| Use ground-truth trajectory information instead of visual inputs | GPT-4o versus DriveMM · STSnu Ego-to-Agent | accuracy: 48.31% vs. 75.75% | Helped |
| Use ground-truth trajectory information instead of visual inputs | GPT-4o versus InternVL 2.5 8B · STSnu Agent-to-Agent | accuracy: 51.77 vs 43.38 | Hurt |
| Use DriveMM rather than the other evaluated driving expert models | Driving expert VLMs · STSnu Average | accuracy | Helped |
Surprising findings
- Text-only LLMs given ground-truth trajectories outperform visual driving expert VLMs by a large margin.
- Off-the-shelf VLMs and top LLMs do better than driving experts on the harder agent-to-agent scenarios.
Limitations
- NuScenes is geographically homogeneous (Boston, Singapore) and contains few dangerous or unexpected behaviors, limiting safety-critical scenarios.
- Heuristic scenario definitions may not capture variable-length spatio-temporal processes such as u-turns.
- Human verifiers disagreed substantially on negative samples (20.8%).
Experimental setup
Datasets: NuScenes validation set
Benchmarks: STSnu
Metrics: accuracy weighted average accuracy across scenario categories
Baselines: Llama 3.2 DeepSeek V3 GPT-4o Qwen2.5-VL 7B InternVL 2.5 1B InternVL 2.5 8B Senna-VLM OmniDrive DriveMM
Builds on: NuScenes DriveLM NuInstruct
Closest papers by meaning
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.92)
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.90)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.90)
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.89)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.89)
- LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.