Research map / Video / Video-language understanding
TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TUMTraffic-VideoQA, a roadside traffic video benchmark unifying multiple-choice video QA, referred object captioning, and spatio-temporal object grounding with tuple-based object expressions, plus a Qwen2-based baseline with multi-resolution visual token sampling.
Problem
Existing traffic vision-language datasets focus on driving ego-vehicle views and single tasks, leaving video-based roadside (third-party perspective) traffic understanding under-explored, and lacking a unified spatio-temporal object reference format.
Approach
A four-element tuple (object id, normalized frame time, x, y) is used as a unified spatio-temporal object expression across QA, captioning, and grounding; a Qwen2-based baseline (TUMTraffic-Qwen) is trained with several multi-resolution visual token sampling strategies (spatial pooling, token pruning, temporal pooling).
Key results
- TUMTraffic-Qwen 7B with MultiRes Temporal-Pooling reaches 81.95% overall multi-choice QA accuracy, versus 56.36% for the best open-source model (LLaVA-OneVision 7B).
- Fine-tuned 0.5B baseline achieves about 0.26 spatio-temporal error on grounding, while zero-shot open-source VLMs range from 0.69 to 0.84.
- Blind (no visual input) baselines still reach high accuracy, suggesting text-only biases in the benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the Qwen-2 baseline from 0.5B to 7B | TUMTraffic-Qwen with MultiRes Temporal-Pooling · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 79.83 -> 81.95 | Helped |
| Use MultiRes Temporal-Pooling instead of Spatial Pooling (ablation) | TUMTraffic-Qwen 0.5B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 78.84 -> 79.83 | Helped |
| Use MultiRes Temporal-Pooling instead of Spatial Pooling (ablation) | TUMTraffic-Qwen 7B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 81.14 -> 81.95 | Helped |
| Use multi-resolution sampling instead of Spatial-Pooling (ablation) | TUMTraffic-Qwen baseline · TUMTraffic-VideoQA validation, Spatio-Temporal Grounding | spatio-temporal error: 0.2382 -> 0.2493 | Hurt |
| Use MultiRes Temporal-Pooling instead of Spatial-Pooling (ablation) | TUMTraffic-Qwen 0.5B · TUMTraffic-VideoQA validation, Referred Object Captioning | CIDEr: 2.5195 -> 2.5464 | Helped |
| Use MultiRes Spatial-Pooling instead of Spatial-Pooling (ablation) | TUMTraffic-Qwen 7B · TUMTraffic-VideoQA validation, Referred Object Captioning | CIDEr: 2.5613 -> 2.6113 | Helped |
| Increase input from no frames to 1 frame (ablation) | TUMTraffic-Qwen 0.5B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 67.59 -> 76.35 | Helped |
| Increase input from 1 frame to 11 frames (ablation) | TUMTraffic-Qwen 0.5B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 76.35 -> 78.97 | Helped |
| Increase input from 11 frames to 101 frames (ablation) | TUMTraffic-Qwen 0.5B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 78.97 -> 78.84 | Hurt |
| Increase input from no frames to 101 frames (ablation) | TUMTraffic-Qwen 7B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: +10.13 | Helped |
| Increase input from 11 frames to 101 frames (ablation) | TUMTraffic-Qwen 7B · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 80.93 -> 81.14 | Helped |
| Remove all visual input (ablation) | TUMTraffic-Qwen baseline · TUMTraffic-VideoQA validation, Multi-Choice QA | overall accuracy: 78.84 -> 67.59; 81.14 -> 71.01 | Hurt |
Surprising findings
- Without any visual input, the fine-tuned baseline still achieves high performance across all three tasks, indicating exploitable text-based biases.
- Multi-resolution strategies improve QA and captioning but slightly degrade spatio-temporal grounding performance.
Limitations
- Zero-shot evaluation and baseline results are reported with limited model variety and no LLM-based caption metrics.
- Spatial reasoning via planar-assumption pseudo-3D projection may be approximate.
- Multi-resolution gains are modest and inconsistent across tasks.
Experimental setup
Datasets: TUMTraffic-VideoQA
Benchmarks: TUMTraffic-VideoQA
Metrics: top-1 accuracy BLEU-4 CIDEr ROUGE-L METEOR SPICE temporal error spatial error spatio-temporal error
Baselines: LLaVA-OneVision Qwen2-VL VideoLLaMA2
Builds on: LLaVA-OneVision Qwen2 SigLIP DriveLM
Closest papers by meaning
- RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives (CVPR 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.87)
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.86)
- iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning (NeurIPS 2025 · similarity 0.86)
- RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.