Research map / Large language models / Agents, code and math
Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments
Junfeng Yan, Biao Wu, Meng Fang, Ling Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Automotive-ENV, a benchmark for evaluating multimodal models on in-vehicle cockpit GUI tasks involving vehicle-state reasoning, implicit intent, and safety-critical decisions.
Problem
Multimodal models are well studied in web, desktop, and mobile GUI environments, but their performance in automotive cockpit systems is largely unexplored, with unique challenges such as vehicle-state reasoning, implicit intent understanding, and safety-critical decision making.
Approach
A benchmark environment specifically for automotive cockpit GUIs that targets vehicle-state reasoning, implicit intent understanding, and safety-critical decisions beyond conventional GUI interaction settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a larger general-purpose multimodal model instead of GUI-specialized VLMs | Automotive-ENV agents · General Explicit | success rate (%): Qwen3-VL-8B 43.7 -> Kimi-K2.5 75.8 | Improved |
| move from General to Safety-Critical tasks | Kimi-K2.5 · Explicit tasks | success rate (%): 75.8 -> 29.7 | Hurt |
| move from explicit to implicit instructions | Kimi-K2.5 · General tasks | success rate (%): 75.8 -> 56.3 | Hurt |
| move from explicit to implicit instructions | Qwen3.5-122B-A10B · General tasks | success rate (%): 70.5 -> 52.2 | Hurt |
| move from explicit to implicit instructions | Gemini 3 Flash · General tasks | success rate (%): 45.6 -> 47.1 | No clear effect |
| GUI-specialized training on web, desktop or mobile data | GUI-specialized VLMs · Automotive-ENV | success rate (%) | Hurt |
| increase model size from 3B to 7B | Qwen2.5-VL · General Explicit | success rate (%): 13.6 -> 26.5 | Improved |
| increase model size from 3B to 7B | JEDI · General Explicit | success rate (%): 16.7 -> 25.2 | No clear effect |
| increase model size from 2B to 8B | MAI-UI · General Implicit | success rate (%): 14.2 -> 32.4 | Improved |
| increase task difficulty from Easy to Hard | Kimi-K2.5, Qwen3.5-122B-A10B, Gemini 3 Flash · Automotive-ENV difficulty groups | success rate | Hurt |
| add vehicle-state querying and safety-aware reasoning to literal instruction following | evaluated agents · Safety-Critical Implicit | success rate (%) | Improved |
Experimental setup
Benchmarks: Automotive-ENV
Closest papers by meaning
- SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models (ICML 2025 · similarity 0.87)
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.84)
- GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents (CVPR 2026 · similarity 0.84)
- Hybrid-Driving: An Autonomous Driving Decision Framework Integrating Large Language Models, Knowledge Graphs and Driving Rules (AAAI 2025 · similarity 0.83)
- DriveCombo: Benchmarking Compositional Traffic Rule Reasoning in Autonomous Driving (CVPR 2026 · similarity 0.83)
- Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
