Research map / Vision-language and multimodal / Human and embodied modeling
MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MPDrive, which overlays detected objects in driving images with numeric visual markers so the MLLM predicts marker indices instead of raw coordinates, plus scene- and instance-level visual prompts, improving AD-VQA spatial understanding on DriveLM and CODA-LM.
Problem
Autonomous driving VQA requires spatial understanding, but MLLMs express object coordinates as text, creating a semantic gap between coordinate text and visual/linguistic descriptions and increasing the expressive burden.
Approach
Replace textual coordinate generation with text-based visual marker prediction: a detection expert (StreamPETR) overlays indexed markers and masks on images; a Marker ControlNet fuses marker and original image features into scene-level prompts, and mask average pooling yields instance-level prompts that are fed to the LLM alongside text tokens.
Key results
- On DriveLM, MPDrive (InternVL-2 base) reaches Match 13.43 vs 7.59 for InternVL-2, with CIDEr 3.56 and METEOR 38.31.
- On CODA-LM, MPDrive achieves a general score of 41.80 vs 34.60 for Qwen-VL-Max, and suggestion score 58.20 vs 47.40 for the best baseline shown.
- Applying MPDrive to LLaMA-Adapter raises Match from 1.48 to 10.05 and BLEU-4 from 45.96 to 47.97.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MPDrive with visual markers and spatial prompts | InternVL-2 · DriveLM | match: 7.59 -> 13.43 | Helped |
| use MPDrive with visual markers and spatial prompts | InternVL-2 · DriveLM | BLEU-4: 51.42 -> 52.71 | Helped |
| use MPDrive with visual markers and spatial prompts | CODA-LM · CODA-LM | General Text-Score: 34.60 -> 41.80 | Helped |
| add Visual Marker (ablation) | InternVL-2 · DriveLM | match: 7.59 -> 11.89 | Helped |
| add Visual Marker (ablation) | InternVL-2 · DriveLM | accuracy: 82.54 -> 80.42 | Hurt |
| add MCNet to Visual Marker (ablation) | InternVL-2 · DriveLM | match: 11.89 -> 9.70 | Hurt |
| add MCNet to Visual Marker (ablation) | InternVL-2 · DriveLM | BLEU-4: 52.04 -> 52.56 | Helped |
| add instance-level visual prompts (ablation) | MPDrive with Visual Marker and MCNet · DriveLM | match: 9.70 -> 13.43 | Helped |
| add instance-level visual prompts (ablation) | MPDrive with Visual Marker and MCNet · DriveLM | accuracy: 78.83 -> 85.18 | Helped |
| apply MPDrive to LLaMA-Adapter (ablation) | LLaMA-Adapter · DriveLM | match: 1.48 -> 10.05 | Helped |
| apply MPDrive to LLaMA-Adapter (ablation) | LLaMA-Adapter · DriveLM | BLEU-4: 45.96 -> 47.97 | Helped |
| apply MPDrive to InternVL-2 (ablation) | InternVL-2 · DriveLM | accuracy: 82.54 -> 85.18 | Helped |
Surprising findings
- Adding visual markers alone raises Match but slightly lowers accuracy (84.66 to 80.42 range in ablation), and adding MCNet reduces Match from 11.89 to 9.70 while improving BLEU-4 and METEOR.
Limitations
- Depends on a prior detection expert; its errors propagate to the output.
- Does not address long-horizon temporal perception.
- Ablation shows trade-offs: marker and MCNet components reduce some metrics (e.g., accuracy, Match) while improving others.
Experimental setup
Datasets: DriveLM CODA-LM
Benchmarks: DriveLM CODA-LM
Metrics: Match Accuracy BLEU-4 ROUGE L CIDEr METEOR CODA-LM regional perception score general score suggestion score
Baselines: DriveLM-Agent EM-VLM4AD MiniDrive LLaMA-Adapter InternVL-2 LLaVA1.5 Qwen-VL-Chat Qwen-VL-Max
Builds on: Set-of-Mark prompting ControlNet LLaMA-Adapter StreamPETR
Closest papers by meaning
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.83)
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · similarity 0.82)
- DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving (CVPR 2024 · similarity 0.82)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.81)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.81)
- Language Prompt for Autonomous Driving (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.