Research map / Vision-language and multimodal / Multimodal LLMs
MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MAPLM, a large-scale real-world vision-language benchmark (MAPLM-QA, 13,775 annotated frames) combining panoramic images, LiDAR BEV images, and HD map text, plus a LLaVA-style instruction-tuned baseline.
Problem
Existing traffic and driving vision-language benchmarks lack multi-modal alignment of point clouds, panoramic images, and HD map text, and are small or built on simulators or re-labeled nuScenes data, limiting their use for evaluating multimodal LLMs in map and traffic scene understanding.
Approach
A real-world dataset collected by HD-map production vehicles pairs panoramic images, LiDAR BEV projections, and HD-map-derived text; a baseline fuses dual CLIP-based encoders with HD-map text into a LLaMA-2/Vicuna LLM, trained in two stages (BEV encoder pretraining on 510K pairs, then LoRA instruction tuning).
Key results
- Baseline with LoRA on Vicuna-7B reaches 85.52 QNS and 57.99 FRM on MAPLM-QA, versus 55.20-ish for zero-shot GPT-4V (56.25 QNS, 18.75 FRM with 0-shot GPT-4V baseline row being 56.25 overall)
- GPT-4V 0-shot achieves 18.75 FRM and 57.81 QNS; LLaVA with Vicuna-7B LoRA reaches 64.89 FRM... (see table) and 77.53 QNS
- Adding LiDAR BEV modality raises baseline FRM from 45.47 to 57.99 and QNS from 80.25 to 85.52
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LoRA to instruction tuning (ablation) | MAPLM baseline with Vicuna-7B · MAPLM-QA test | QNS: 82.13 -> 85.52 | Helped |
| Add LoRA to instruction tuning (ablation) | MAPLM baseline with LLaMA-2-7B · MAPLM-QA test | QNS: 76.38 -> 81.65 | Helped |
| Add panoramic image input alongside point cloud input (ablation) | MAPLM baseline with Vicuna-7B · MAPLM-QA test | QNS: 80.25 -> 85.52 | Helped |
| Add panoramic image input alongside point cloud input (ablation) | GPT-4V 0-shot · MAPLM-QA test | QNS: 53.28 -> 57.81 | Helped |
| Use Vicuna-7B rather than LLaMA-2-7B as the LLM backbone | MAPLM baseline IT+LoRA · MAPLM-QA test | QNS: 81.65 -> 85.52 | Helped |
Surprising findings
- GPT-4V zero-shot and few-shot inference performs poorly on MAPLM-QA, especially in counting lanes, despite good results on nuScenes traffic scenes
- Initial LLM weights (Vicuna vs LLaMA-2) substantially affect driving/map QA performance
Limitations
- GPT-4V shows lane hallucination and low frame-level accuracy on the benchmark
- Baseline does not exceed 60% FRM, leaving room for improvement
- Authors do not aim to surpass state-of-the-art multimodal LLMs; baseline is intended for consistent benchmarking
Experimental setup
Datasets: MAPLM-QA MAPLM image-text pairs
Benchmarks: MAPLM-QA
Metrics: FRM QNS LAN DES INT QLT SCN
Baselines: GPT-4V LLaVA
Builds on: LLaVA InstructBLIP LoRA
Closest papers by meaning
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.90)
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.90)
- Embodied Scene Understanding for Vision Language Models via MetaVQA (CVPR 2025 · similarity 0.90)
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.90)
- RoadSceneBench: A Lightweight Benchmark for Mid-Level Road Scene Understanding (CVPR 2026 · similarity 0.89)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.