Research map / Vision-language and multimodal / Visual question answering
NuScenes-QA: A Multi-Modal Visual Question Answering Benchmark for Autonomous Driving Scenario
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NuScenes-QA, a template-generated multi-modal, multi-frame outdoor VQA benchmark (460K QA pairs over 34K scenes) built from nuScenes 3D annotations, with 3D-detector-plus-QA-head baselines.
Problem
Existing VQA benchmarks are single-modal, single-frame, and indoor, so they do not capture the multi-modal (camera + LiDAR), multi-frame, outdoor conditions of autonomous driving scenes.
Approach
Automatically generate QA pairs for nuScenes keyframes by converting 3D detection annotations into scene graphs (with spatial relations computed from BEV box angles relative to the ego car) and instantiating manually designed CLEVR-style question templates, then use detected object features cropped from BEV maps as inputs to an MCAN answer head.
Key results
- Best camera+LiDAR baseline MSMDFusion+MCAN reaches 60.4% overall accuracy, versus 53.4% for the question-only blind model and 84.3% for ground-truth-label MCAN.
- Counting is the hardest question type; the best baseline reaches only 23.2% on counting.
- Adding detected boxes slightly lowers accuracy (59.5% to 58.9%) for CenterPoint+MCAN, while ground-truth boxes raise it from 70.8% to 84.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use visual input instead of question-only input | MSMDFusion+MCAN vs Q-Only · NuScenes-QA test | accuracy: 53.4% -> 60.4% | Helped |
| use camera-LiDAR fusion instead of camera-only input | MCAN baselines · NuScenes-QA test | accuracy: 57.9 -> 60.4 | Helped |
| use MCAN instead of BUTD as the QA head | CenterPoint baseline · NuScenes-QA test | accuracy: 1.4% | Helped |
| add detected box features (ablation) | CenterPoint+MCAN · NuScenes-QA test | accuracy: 59.5 -> 58.9 | Hurt |
| add ground-truth box features (ablation) | GroundTruth+MCAN · NuScenes-QA test | accuracy: 70.8% to 84.3% | Helped |
| use rotated-box crop instead of circumscribed axis-aligned crop (ablation) | CenterPoint+MCAN · NuScenes-QA test | accuracy: 58.8 -> 59.5 | Helped |
| use mean pooling instead of max pooling (ablation) | CenterPoint+MCAN · NuScenes-QA test | accuracy: 58.9% to 59.5% | Helped |
Surprising findings
- Adding detected 3D box features slightly hurts performance, while adding ground-truth box features substantially helps.
- LiDAR-only CenterPoint outperforms camera-only BEVDet on this benchmark.
Limitations
- Questions focus on foreground objects and spatial relations, ignoring background and semantic relations.
- Template-based language lacks diversity and reasoning depth compared with human-written questions.
- Detected-box features are noisy and hurt the QA model, indicating immature 3D detection for this task.
Experimental setup
Datasets: nuScenes CLEVR-style generated NuScenes-QA
Benchmarks: NuScenes-QA
Metrics: top-1 accuracy
Baselines: Q-Only BEVDet+BUTD CenterPoint+BUTD MSMDFusion+BUTD BEVDet+MCAN CenterPoint+MCAN MSMDFusion+MCAN GroundTruth+BUTD GroundTruth+MCAN
Builds on: CLEVR MCAN BUTD CenterPoint BEVDet MSMDFusion
Closest papers by meaning
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.88)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA (AAAI 2024 · similarity 0.88)
- nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving (NeurIPS 2026 · similarity 0.87)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.