Research map / Vision-language and multimodal / Multimodal LLMs
Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NuInstruct, a 91K multi-view video-QA driving dataset across 17 subtasks generated via an SQL-based pipeline over nuScenes, and BEV-InMLLM, which injects instruction-aware BEV features into existing MLLMs (BLIP-2, MiniGPT-4, Video-LLaMA).
Problem
Existing language-based driving benchmarks cover only a subset of driving tasks and use single-view, image-based inputs, omitting temporal, multi-view, and spatial information needed for holistic autonomous driving understanding.
Approach
Generate instruction-response pairs automatically with task-specific SQL queries over a scene database built from nuScenes annotations, and inject instruction-aware BEV tokens (from a frozen BEV extractor via a BEV Q-Former and cross-attention injection) into a multi-view MLLM that also takes video input.
Key results
- BEV-InMLLM improves over base models on NuInstruct, e.g., MiniGPT-4 distance MAE drops from 30.2 to 23.6 and planning-with-reasoning BLEU rises from 26.3 to 35.6.
- MV-MLLM substantially improves risk and planning tasks by about 5% and 6% over single-view baselines, per the authors.
- Ablation on MiniGPT-4 shows removing BEV injection lowers the whole-task score from 22.2 to 20.1, and removing video lowers it to 21.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add video input for temporal cues (ablation) | BEV-InMLLM · NuInstruct | Temporal score (lower-is-better column): 7.4 -> 3.7 | Helped |
| add multi-view information (ablation) | BEV-InMLLM · NuInstruct | Multi-view score (lower-is-better column): 6.0 -> 3.8 | Helped |
| add BEV injection module (ablation) | BEV-InMLLM · NuInstruct | Spatial score (lower-is-better column): 18.0 -> 13.9 | Helped |
| add BEV injection module (ablation) | BEV-InMLLM · NuInstruct | Whole score: 20.1 -> 22.2 | Helped |
| add instruction tokens to the BEV Q-Former (ablation) | BEV-InMLLM · NuInstruct | Whole score: 20.1 -> 21.5 | Helped |
| increase BEV query count from 16 to 32 (ablation) | BEV-InMLLM · NuInstruct | Spatial score (lower-is-better column): 14.5 -> 13.9 | Helped |
| increase BEV query count from 32 to 64 (ablation) | BEV-InMLLM · NuInstruct | Whole score: 21.5 -> 21.6 | Helped |
| inject instruction-aware BEV features into the multi-view Q-Former output rather than filtered multi-view tokens (ablation) | BEV-InMLLM · NuInstruct | Whole score: 20.8 -> 21.5 | Helped |
| use BEVFusion rather than LSS as the BEV extractor (ablation) | BEV-InMLLM · NuInstruct | Spatial score (lower-is-better column): 13.9 -> 13.2 | Helped |
| add MV-MLLM modules to MiniGPT-4 | MiniGPT-4 · NuInstruct | Planning with Reasoning BLEU: 26.3 -> 32.3 | Helped |
| inject BEV into MiniGPT-4 MV-MLLM | MiniGPT-4 MV-MLLM · NuInstruct | Planning with Reasoning BLEU: 32.3 -> 35.6 | Helped |
| inject BEV into MiniGPT-4 MV-MLLM | MiniGPT-4 MV-MLLM · NuInstruct | Instance number MAE: 4.1 -> 3.8 | Hurt |
Limitations
- Dataset lacks traffic light information.
- Dataset lacks 3D object detection tasks.
- Responses are generated from templates or GPT-4 rewriting rather than human-written language.
Experimental setup
Datasets: nuScenes NuInstruct
Benchmarks: NuInstruct
Metrics: MAE accuracy MAP BLEU
Baselines: BLIP-2 MiniGPT-4 Video-LLaMA MV-MLLM
Builds on: BLIP-2 MiniGPT-4 Video-LLaMA LSS BEVFormer ST-Adapter BLIP-2 Q-Former
Closest papers by meaning
- World Knowledge-Enhanced Reasoning Using Instruction-Guided Interactor in Autonomous Driving (AAAI 2025 · similarity 0.89)
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.89)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.88)
- Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.88)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.