Research map / Vision-language and multimodal / Robot action and navigation

From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving

NeurIPS 2026 · Vision-language and multimodal › Robot action and navigation · approach: Vision-language-action foundation models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compares VLM-hidden and vision-only encoders under a shared diffusion planner on NAVSIM, finds shared-plus-unique representation structure, and turns branch complementarity into HybridDriveVLA (PDMS 92.10) and DualDriveVLA (fast-slow, 91.00 PDMS at ~1.9x lower latency).

Problem

Whether VLM and vision-only visual encoders differ in representation and behavior after policy learning in end-to-end driving, and whether those differences can be exploited for planning accuracy and latency.

Approach

Treat VLM-hidden and vision-only diffusion policies as complementary candidate generators: analyze shared vs. unique latents with a Shared-Unique SAE, then select among cross-model trajectories with a learned trajectory scorer, and use a fast-slow policy that invokes the VLM only when needed.

Key results

• HybridDriveVLA improves NAVSIM PDMS from 90.80 (InternVL3-2B baseline) to 92.10.
• DualDriveVLA invokes the VLM in about 15% of scenarios, reaching 91.00 PDMS with latency about 150 ms vs 280 ms for the VLM baseline (~1.9x).
• Oracle best-of-two (VLM+ViT) reaches 93.58 PDMS; representation-only gates reach at most 90.96.

Surprising findings

• Representation-only gating based on alignment or latent statistics barely beats the VLM baseline (90.80 to 90.96) despite a large oracle gap.
• Vision-only encoders win more often in simple lane-keeping scenes while VLMs dominate complex semantic scenes.

Limitations

• Comparison is not upstream-pretraining-budget symmetric: VLM branch gets driving-domain adaptation while ViT does not in main setting.
• SAE is not a full causal disentanglement method, so latent-intervention evidence is not causal proof.
• Analysis focuses on native-VLM-encoder VLA systems, not LLM-centric architectures.

Experimental setup

Datasets: NAVSIM nuPlan (AsyncDrive stack)

Benchmarks: NAVSIM navtest (PDMS, NAVSIM-v1) NAVSIM-v2 navtest (EPDMS) nuPlan (AsyncDrive)

Metrics: PDMS EPDMS CKA CCA R^2 cross reconstruction latency

Baselines: InternVL3-2B (RecogDrive-style VLM) ViT-large ResNet-50/101 EVA-CLIP02-base Trajectory mean Rule-based selection Random Forest gate DiffusionDriveV2 ReCogDrive WAM-diff DriveVLA-W0 AutoVLA

Builds on: RecogDrive DiffusionDrive DrivoR VICReg AsyncDrive

Closest papers by meaning

• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning (AAAI 2026 · similarity 0.92)
• $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.91)
• LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving (NeurIPS 2026 · similarity 0.90)
• Evaluating Vision-Language Models as Evaluators in Path Planning (CVPR 2025 · similarity 0.90)
• SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving (CVPR 2025 · similarity 0.90)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation (NeurIPS 2026 · similarity 0.90)