Research map / Vision-language and multimodal / Multimodal LLMs
WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WOMD-Reasoning, a ~3M-QA multimodal language dataset on Waymo Open Motion Dataset focused on traffic-rule and intention-induced interactions, plus Motion-LLaVA, a LLaVA-based model with a MultiPath++ motion encoder fine-tuned on it.
Problem
Existing driving language datasets focus on proximity-based interactions and rarely cover non-proximate interactions induced by traffic rules or human intentions, leaving LLMs weak at interaction analysis.
Approach
Automatically generate interaction Q&As for WOMD scenes by translating motion data into rule-based text descriptions and prompting GPT-4 Turbo with chain-of-thought question structures covering traffic rules and intentions; then fine-tune a LLaVA model with a motion encoder (Motion-LLaVA) on the resulting data, and show that its outputs can improve trajectory prediction.
Key results
- Motion-LLaVA reaches ROUGE 0.792, BLEU 0.616, CIDEr 5.69 and GPT Score 7.02 overall, beating fine-tuned LLaVA (0.779, 0.581, 5.51, 6.88).
- On factual questions, Motion-LLaVA reaches 56.2% Acc@1.0m versus 10.9% for fine-tuned LLaVA, with median error 1.0 vs 5.0.
- Adding Motion-LLaVA interaction outputs to MultiPath++ lowers minFDE6 from 1.27 to 1.18 and MR6 from 12.59 to 11.69 (about -7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | ROUGE: 0.779 -> 0.792 | Helped |
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | BLEU: 0.581 -> 0.616 | Helped |
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | METEOR: 0.439 -> 0.449 | Helped |
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | CIDEr: 5.51 -> 5.69 | Helped |
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | SPICE: 0.735 -> 0.744 | Helped |
| Use motion data as input instead of BEV plots | Fine-tuned LLaVA · All questions | GPT Score: 6.88 -> 7.02 | Helped |
| Fine-tune LLaVA on WOMD-Reasoning | LLaVA-v1.5-7B · All questions | GPT Score: 2.31 -> 6.88 | Helped |
| Use motion data instead of BEV plots | Fine-tuned LLaVA · Facts only | Acc₁.₀: 10.9% -> 56.2% | Helped |
| Use motion data instead of BEV plots | Fine-tuned LLaVA · Facts only | Median Error: 5.0 -> 1.0 | Helped |
| Add Motion-LLaVA interaction outputs | Multipath++ · Vehicle trajectory prediction | minFDE₆: 1.27 -> 1.18; -7.35% | Helped |
| Add Motion-LLaVA interaction outputs | Multipath++ · Vehicle trajectory prediction | MR₆: 12.59 -> 11.69; -7.10% | Helped |
Limitations
- Dataset answers are generated by GPT-4-Turbo from rule-based text translations, with human-estimated accuracy of about 92% (only 1,610 Q&As checked by 4 people).
- Only scenes with objects_of_interest labels were translated, so coverage is restricted to confirmed interactive scenarios.
- MultiPath++ code and checkpoints were not released, so the authors used a third-party reimplementation.
Experimental setup
Datasets: Waymo Open Motion Dataset
Benchmarks: WOMD-Reasoning
Metrics: ROUGE-L BLEU-4 METEOR CIDEr SPICE GPT Score Acc@1.0m Median Error minFDE6 MR6
Baselines: Non-Fine-tuned LLaVA Fine-tuned LLaVA Shuffled Ground-truths Multipath++
Builds on: LLaVA MultiPath++ ScenarioNet Chain-of-Thought prompting
Closest papers by meaning
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.88)
- Trajeglish: Traffic Modeling as Next-Token Prediction (ICLR 2024 · similarity 0.86)
- nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving (NeurIPS 2026 · similarity 0.86)
- PSI: A Benchmark for Human Interpretation and Response in Traffic Interactions (NeurIPS 2025 · similarity 0.86)
- LMDrive: Closed-Loop End-to-End Driving with Large Language Models (CVPR 2024 · similarity 0.85)
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.