Research map / Vision-language and multimodal / Multimodal LLMs
SURDS: Benchmarking Spatial Understanding and Reasoning in Driving Scenarios with Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SURDS, a nuScenes-based VQA benchmark for spatial reasoning in driving scenes, and improves Qwen2.5-VL-3B with SFT on CoT data followed by GRPO using location and logic rewards.
Problem
Vision language models lack reliable fine-grained spatial understanding (orientation, depth, pixel localization, pairwise distance, lateral and front-behind relations) in outdoor driving scenes, and no benchmark systematically evaluates this.
Approach
A driving-specific spatial VQA benchmark built from nuScenes with six task types, plus a GRPO alignment scheme using a location (bbox IoU) reward and a self-verified logic reward computed from the model's own reasoning trace, added to format and accuracy rewards, applied after SFT on QVQ/Qwen-generated CoT data.
Key results
- The GRPO-aligned Qwen2.5-VL-3B (SFT-GRPO-LocLogic) reaches an overall SURDS score of 40.80, versus 33.47 for Qwen2.5-VL-72B-Instruct and 13.30 for GPT-4o.
- SFT alone yields a 26.94 score; adding format, location, accuracy and logic rewards in GRPO raises it to 40.80.
- Location and logic rewards together give the largest gain; each alone gives only marginal improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply GRPO to Qwen2.5-VL-3B without SFT (ablation) | Qwen2.5-VL-3B · SURDS | Score: 13.48 -> 24.30 | Helped |
| add location and logic rewards to GRPO without SFT (ablation) | Qwen2.5-VL-3B-GRPO · SURDS | Score: 24.30 -> 24.50 | No clear effect |
| apply SFT to the base Qwen2.5-VL-3B model (ablation) | Qwen2.5-VL-3B · SURDS | Score: 13.48 -> 26.94 | Helped |
| apply GRPO after SFT using format and accuracy rewards (ablation) | Qwen2.5-VL-3B-SFT · SURDS | Score: 26.94 -> 29.66 | Helped |
| add location and logic rewards to GRPO after SFT (ablation) | Qwen2.5-VL-3B-SFT-GRPO · SURDS | Score: 29.66 -> 40.80 | Helped |
| add location reward without logic reward to GRPO after SFT (ablation) | Qwen2.5-VL-3B-SFT-GRPO · SURDS | Score: 29.66 -> 31.53 | Helped |
| add logic reward without location reward to GRPO after SFT (ablation) | Qwen2.5-VL-3B-SFT-GRPO · SURDS | Score: 29.66 -> 29.73 | No clear effect |
| use {-1, 1} rather than {0, 1} binary reward values (ablation) | Qwen2.5-VL-3B-SFT-GRPO-LocLogic · SURDS | Score: 40.80 -> 31.55 | Hurt |
| use SFT-GRPO with location and logic rewards instead of Gemini-2.0-flash | SURDS evaluation · SURDS | Score: 35.71 -> 40.80 | Helped |
Surprising findings
- Larger models do not reliably outperform smaller ones on spatial tasks; parameter count is not a reliable predictor of spatial competence.
- A {-1,1} reward setting underperforms the {0,1} setting (31.55 vs 40.80), attributed to negative-reward gradient saturation.
Limitations
- Method only tested on a 3B model; larger variants untested.
- Benefits of linear reward scaling and multi-stage GRPO schedules unclarified.
- Pairwise front-behind performance drops after alignment (8.54 vs 21.62 SFT baseline in ablation).
Experimental setup
Datasets: nuScenes
Benchmarks: SURDS
Metrics: accuracy centerness-based localization score
Baselines: GPT-4o GPT-4o-mini Gemini-1.5-pro Gemini-2.0-flash Qwen2.5-VL-72B-Instruct SpatialBot SpatialRGPT LLaVA-OV-Qwen2-72b-si
Builds on: GRPO Embodied-R Qwen2.5-VL
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.90)
- GTR-Bench: Evaluating Geo-Temporal Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.