Research map / Vision-language and multimodal / Multimodal LLMs

DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 17 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DriveGPT4-V2, a small-LLM (0.5B) multimodal planner that maps three front-view camera images and speed to low-level driving controls in closed loop, trained with behavior cloning plus DAgger-style on-policy supervision from a privileged-information LLM expert, reaching SOTA DS on CARLA Longest6.

Problem

Closed-loop end-to-end autonomous driving with multimodal LLMs has been studied mostly open-loop; behavior cloning suffers from error accumulation in closed loop, and LLM planners are slow.

Approach

Combines a multi-view visual tokenizer over three front cameras, MLP decision heads that output numeric decisions from single tokens (instead of text numbers), PID conversion to controls, and an LLM expert with privileged object information that supervises the student online via DAgger-style aggregation.

Key results

• DriveGPT4-V2 reaches 70 Driving Score, 91 Route Completion and 0.77 Infraction Score on CARLA Longest6, above the best baseline Transfuser++ at 65 DS.
• Under matched training data, Transfuser++ gets 58 DS while DriveGPT4-V2 gets 70 DS with expert supervision; ablation shows expert supervision adds 7 DS (63 to 70).
• Using 0.5B LLM runs at 8.1 FPS versus 0.4 FPS for an 8B LLaVA-LLaMA3.1 model with similar DS (65).

Measured effects reported in the paper

Surprising findings

• Scaling the LLM from 0.5B to 8B did not improve driving score (63 vs 65 DS) while slowing inference more than 10x.
• Using predicted waypoints for PID control performed worse than using target speed and route points.

Limitations

• Uses only tiny LLMs (0.5B-1.5B); larger LLMs were not explored due to compute and speed constraints.
• Evaluated only in CARLA simulation; no real-world deployment.
• Expert relies on privileged ground-truth information unavailable at inference.

Experimental setup

Datasets: CARLA autopilot driving data (Transfuser++ pipeline)

Benchmarks: CARLA Longest6

Metrics: Driving Score Route Completion Infraction Score infraction ratios FPS

Baselines: WOR LAV v1 LAV v2 Interfuser TransFuser Perception PlanT Transfuser++ LMDrive

Builds on: DriveGPT4 LBC LLaVA Transfuser++ MiniGPT4

Closest papers by meaning

• S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.89)
• MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.89)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.89)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.88)
• CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model (NeurIPS 2025 · similarity 0.88)
• Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.88)