Research map / Vision-language and multimodal / Multimodal LLMs
World Knowledge-Enhanced Reasoning Using Instruction-Guided Interactor in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an instruction-guided interactor that selects top-k instruction-relevant visual/BEV tokens and pre-fuses them via cross-attention, plugged into an MLLM (LLaMA3-8B) for multi-view driving reasoning under occlusion, plus a 200K-QA object-level risk assessment dataset.
Problem
MLLM-based autonomous driving reasons well in visible regions but struggles with perception-limited (occluded) areas and multi-view video inputs whose long token sequences hinder integration of perception with world knowledge.
Approach
An instruction-guided interactor selects the top-k visual and BEV tokens most similar to the language instruction and refines them via cross-attention over the full features before feeding a compact sequence to the LLM; paired with a three-stage training pipeline and a new object-level risk assessment (ORA) dataset.
Key results
- On NuScenes-MQA, the model reports ACC 74.4 and BLEU4 48.6 versus OPT-1.3B baselines (ACC 60.4), a claimed 10.6% improvement in ACC over prior SoTA.
- On OmniDrive-NuScenes, CIDEr rises to 103.9 from 68.6 (claimed 51.4% improvement).
- On NuScenes-QA, the model reaches 58.4 ACC, below MSMDFusion+MCAN (60.4), so it is only described as comparable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the instruction-guided interactor (ablation) | Full method · ORA | CIDEr: 313.43 -> 344.55 | Helped |
| add top-k selection (ablation) | Interactor method · ORA | CIDEr: 331.88 -> 344.55 | Helped |
| add top-k selection (ablation) | Interactor method · NuInstruct | ACC: 43.81 -> 52.71 | Helped |
| use the three-stage training strategy (ablation) | Base model · OmniDrive-NuScenes / NuScenes-MQA / NuInstruct ablation | ACC: 64.2 -> 71.9 | Helped |
| add BEV representations (ablation) | Three-stage-trained model · OmniDrive-NuScenes / NuScenes-MQA / NuInstruct ablation | ACC: 71.9 -> 73.3 | Helped |
| add the interactor (ablation) | Model with BEV representations · OmniDrive-NuScenes / NuScenes-MQA / NuInstruct ablation | ACC: 73.3 -> 74.1 | Helped |
| add selection to the interactor (ablation) | Interactor model · OmniDrive-NuScenes / NuScenes-MQA / NuInstruct ablation | ACC: 74.1 -> 74.4 | Helped |
| add single-view pretraining, multi-view pretraining, interactor, and selection (ablation) | Fine-tuned model · OmniDrive | CIDEr: 70.2 -> 103.9 | Helped |
| mask 50% of sky-region visual tokens (ablation) | Full method · NuInstruct | BLUE: 69.85 -> 63.65 | Hurt |
| set k to 30 instead of 90 (ablation) | Instruction-guided interactor · NuInstruct | BLUE4: 16.9 -> 14.2 | Hurt |
| set k to 60 instead of 90 (ablation) | Instruction-guided interactor · NuInstruct | BLUE4: 16.9 -> 13.1 | Hurt |
| set k to 120 instead of 90 (ablation) | Instruction-guided interactor · NuInstruct | BLUE4: 16.9 -> 15.3 | Hurt |
Surprising findings
- Single-view pre-training contributes little to language scores, though it helps grounding.
- Masking up to 30% of sky-region visual tokens barely changes performance, suggesting substantial token redundancy.
Limitations
- Not evaluated on closed-loop benchmarks such as CARLA or NuPlan.
- Not validated on 3D grounding because precise camera parameters are hard to tokenize.
- Reported comparisons are mixed: lower than baseline on NuScenes-QA ACC and some metrics, and the BEV encoder was retrained without future frames and validation data, so baseline numbers are not directly comparable.
Experimental setup
Datasets: NuScenes NuScenes-QA NuScenes-MQA OmniDrive-NuScenes NuInstruct Object-level Risk Assessment (ORA)
Benchmarks: ORA NuScenes-MQA OmniDrive-NuScenes NuInstruct NuScenes-QA nuScenes open-loop planning
Metrics: BLEU ROUGE_L CIDEr METEOR accuracy mAP MAE L2 error collision rate
Baselines: OPT-1.3B OPT-6.7B OmniDrive BEV-InMLLM Video-LLaMA Bunny-Llama3 MSMDFusion+MCAN CenterPoint+MCAN UniAD VAD-Base BEV-Planner
Builds on: Q-Former LLaVA-NeXT SparseBEV Grounding-DINO EVA-02
Closest papers by meaning
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.89)
- Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.88)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.88)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.88)
- AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.