Research map / Vision-language and multimodal / Visual question answering
Embodied Scene Understanding for Vision Language Models via MetaVQA
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MetaVQA, a multiple-choice VQA benchmark with Set-of-Mark-annotated real and simulated driving images plus closed-loop MetaDrive evaluation, showing fine-tuning VLMs on its 150K training questions improves VQA accuracy and closed-loop driving.
Problem
Existing driving VQA benchmarks use incommensurable referral conventions and are evaluated mostly open-loop, so zero-shot evaluation of general-purpose VLMs on embodied scene understanding is hard and safety-critical closed-loop testing is lacking.
Approach
Generate a large multiple-choice VQA corpus from scene graphs of nuScenes and Waymo traffic (rendered in MetaDrive) with Set-of-Mark visual referral, then fine-tune VLMs on it and test them as closed-loop driving agents, including safety-critical CAT scenarios.
Key results
- Fine-tuned InternVL2-8B reaches 0.869 overall VQA test accuracy versus 0.592 zero-shot, the best among evaluated models.
- Training only on simulated images raises InternVL2-8B real-image accuracy from 0.632 to 0.819, showing sim-to-real transfer.
- In closed-loop driving, fine-tuned InternVL2-8B reaches 0.820 test accuracy and lower ADE/FDE than its zero-shot version (13.109/27.873 vs 13.361/30.520), though collision rate rises (0.325 to 0.367).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InternVL2-8B on 150,000 MetaVQA questions | InternVL2-8B · MetaVQA test set | accuracy: 0.592 -> 0.869 | Helped |
| Fine-tune Qwen2 on MetaVQA training data | Qwen2 · MetaVQA test set | accuracy: 0.539 -> 0.844 | Helped |
| Fine-tune Llama3.2 on MetaVQA training data | Llama3.2 · MetaVQA test set | accuracy: 0.500 -> 0.774 | Helped |
| Train on simulated observations only instead of zero-shot evaluation (ablation) | InternVL2-8B · MetaVQA test set | accuracy: 0.592 -> 0.807 | Helped |
| Train on real observations only instead of zero-shot evaluation (ablation) | InternVL2-8B · MetaVQA test set | accuracy: 0.592 -> 0.825 | Helped |
| Combine simulated and real observations for training (ablation) | InternVL2-8B · MetaVQA test set | accuracy: 0.825 -> 0.869 | Helped |
| Increase MetaVQA training size from 9,375 to 37,500 questions (ablation) | InternVL2-8B · MetaVQA test set | accuracy: 0.794 -> 0.845 | Helped |
| Increase MetaVQA training size from 37,500 to 150,000 questions (ablation) | InternVL2-8B · MetaVQA test set | accuracy: 0.845 -> 0.869 | Helped |
| Fine-tune on MetaVQA VQA task | InternVL2-8B · Closed-loop driving scenarios | route completion: 0.637 -> 0.657 | Helped |
| Fine-tune on MetaVQA VQA task | Llama3.2 · Closed-loop driving scenarios | collision rate: 0.483 -> 0.267 | Helped |
| Fine-tune on MetaVQA VQA task | Qwen2 · Closed-loop driving scenarios | route completion: 0.615 -> 0.667 | Helped |
| Fine-tune on MetaVQA VQA task | InternVL2-8B · Closed-loop driving scenarios | collision rate: 0.325 -> 0.367 | Hurt |
Surprising findings
- Training on only simulated observations substantially improves real-image VQA accuracy (0.632 to 0.819), and real-only training also improves sim accuracy.
- Improvements in closed-loop collision rate were not consistent across models, with some fine-tuned models showing higher collision rates.
Limitations
- Only single-frame image observations, no multi-step history.
- Single-camera perspective only, no multi-camera input.
- Some models (LLaVA-NeXT) fail to produce valid answer format, and closed-loop collision results are inconsistent.
Experimental setup
Datasets: nuScenes Waymo Open Motion Dataset MetaVQA
Benchmarks: MetaVQA test set closed-loop MetaDrive driving scenarios
Metrics: accuracy parse failure rate route completion off-road rate collision rate ADE FDE
Baselines: random brake straight LLaVA-NeXT LLaVA-OneVision GPT-4o
Builds on: Set-of-Mark prompting MetaDrive ScenarioNet CAT
Closest papers by meaning
- DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.90)
- MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.90)
- Multi-modal Situated Reasoning in 3D Scenes (NeurIPS 2024 · similarity 0.90)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.89)
- Spatial Reasoning with Vision-Language Models in Ego-Centric Multi-View Scenes (ICLR 2026 · similarity 0.89)
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.