Research map / Large language models / Agents, code and math

CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model

NeurIPS 2025 · Large language models › Agents, code and math · approach: LLM-driven embodied and driving agents · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoC-VLA, an adversarial sim-to-real transfer framework where a simulation-trained teacher VLM and a real-data student VLM (both CoC VLMs built on LLaVA-7b) are aligned via a discriminator, evaluated on nuScenes-VLM.

Problem

Autonomous driving VLMs trained on real-world data miss rare long-tail cases, while simulation-trained models suffer a domain gap and cannot be deployed directly; the paper targets transferring long-tail handling from simulation to real-world driving with explainable chain-of-causality answers.

Approach

Teacher and student CoC VLMs share an architecture, with a lightweight text adapter that feeds the previous frame's simplified answer into the next prompt; a transformer discriminator on their features is trained adversarially in two steps, with gradients passed through the discriminator to update the student VLM.

Key results

• On nuScenes-VLM VQA, the proposed LLaVA-7b model reaches BLEU-4 58.84, ROUGE-L 71.70, CIDEr 23.21 and SPICE 51.88, versus DriveLM's 55.13, 68.32, 3.62 and 48.95 (reproduced).
• Open-loop planning on nuScenes-VLM gives ADE 1.44 and collision rate 1.58%, improving on the FinetuneTwice variant (1.60, 1.73) and on DriveLM (1.71, 1.87).
• Training the teacher only on simulation and testing on real-world data gives ADE 17.33, showing a large sim-to-real gap before adversarial transfer.

Measured effects reported in the paper

Surprising findings

• Mixing simulation and real-world data for the student slightly degraded performance (ADE 1.84 vs 1.66 for real-only training).
• A straight-line dummy regressor was used to check how much of the challenging-split gains could come from easy cases; the proposed model still improved on the student baseline (ADE 1.73 vs 2.08) on the challenging split.

Limitations

• Relies on LLaVA-7b, which the authors note has high inference cost and is hard to deploy online.
• Maintaining two baseline models and a multi-step adversarial training process raises training cost and optimization difficulty.
• Evaluated only on real-world open-loop data; closed-loop metrics not assessed.

Experimental setup

Datasets: nuScenes nuScenes-VLM CARLA-VLM

Benchmarks: nuScenes-VLM

Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L CIDEr GPT score Accuracy Match SPICE

Baselines: DriveLM LMDrive Straight Regressor Baseline Proposed-Mix Proposed-FinetuneTwice

Builds on: LLaVA-v1.5 DriveLM LMDrive VAD

Closest papers by meaning

• AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking (ICLR 2026 · similarity 0.89)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.89)
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.89)