Research map / Video / Video-language understanding
RoadSocial: A Diverse VideoQA Dataset and Benchmark for Road Event Understanding from Social Video Narratives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoadSocial, a 13.2K-video, 260K-QA dataset of road events built from social media videos and comments via a semi-automatic Text/Video LLM pipeline, and benchmarks 18 Video LLMs on it.
Problem
Existing road event video datasets suffer from regional bias, dashcam-only viewpoints, expert-driven annotation, and lack tests of hallucination robustness for Video LLMs.
Approach
Generate QA pairs from social media videos paired with tweet discourse using a hybrid pipeline (segment captions from a Video LLM, summary and QA generation by a Text LLM, refinement, and expert verification), adding adversarial and incompatible (non-road-event) QA to test hallucination robustness.
Key results
- Tarsier-34B achieves the highest overall score of 63.5 across all QA tasks among open-source models, while GPT-4o is the top closed-source performer.
- Most models score below 1% mAP on Temporal Grounding; Gemini-1.5-Pro reaches 18.6% and LLaVA-Video-72B 9.94% among open-source models.
- Fine-tuning LLaVA-OV-7B on the RoadSocial training split yields a 19.4% jump in overall average score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA-OV-7B on RoadSocial training data | LLaVA-OV-7B · RoadSocial-QA test | Overall score (ALL): 50.0 -> 69.4 | Helped |
Surprising findings
- Most Video LLMs score below 1% mAP on temporal grounding, indicating severe temporal localization failures.
- Most models struggle on incompatible (non-road-event) QAs, hallucinating answers for irrelevant videos.
Limitations
- Reliance on social media data may skew coverage toward regions with higher social media use.
- Generic vs specific QA gap suggests models struggle to infer context independently.
- Language diversity is limited in the current dataset.
Experimental setup
Datasets: RoadSocial
Benchmarks: RoadSocial-QA
Metrics: GPT-3.5 score mAP@0.3:0.7
Baselines: GPT-4o Gemini-1.5-Pro InternVL2 Qwen2-VL LLaVA-Video LLaVA-OV Tarsier IXC-2.5 Dolphin VITA ARIA MiniCPM-V 2.6
Builds on: LLaVA-OneVision GPT-4o Tarsier
Closest papers by meaning
- TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes (ICML 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.84)
- OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding? (CVPR 2025 · similarity 0.83)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.83)
- SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models (NeurIPS 2025 · similarity 0.83)
- WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.