Research map / Label-efficient and robust learning / Recognition and long-tail
Embodied Cognition Augmented End2End Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E3AD, which contrastively aligns a video encoder (Driving-Thinking model) with frozen LaBraM EEG features collected from drivers, then injects the frozen cognitive features into end-to-end driving planners to improve open- and closed-loop planning.
Problem
End-to-end autonomous driving planners rely on visual encoders trained only under label supervision, which limits generality; the paper asks whether human driving cognition captured by EEG can provide broader supervision to improve planning.
Approach
Contrastively align a spatio-temporal video encoder with a frozen EEG foundation model (LaBraM) on paired driver video-EEG data, then freeze this Driving-Thinking model and fuse its cognitive features into existing E2E planners (VAD, UniAD, GenAD, LAW) via three designed frameworks, using only standard driving data at inference.
Key results
- E3AD(UniAD) reduces average collision rate from 0.31% to 0.23% and E3AD(VAD-Base) reaches 0.18% average collision on nuScenes open-loop.
- On Bench2Drive, E3AD(UniAD-Base) raises driving score from 45.81 to 50.07 and success rate from 16.36% to 20.12%.
- E3AD(VAD) reaches driving score 47.63 and success rate 19.54% versus 42.35 and 15.00% for VAD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add E³AD to VAD-Base | VAD-Base · nuScenes open-loop | Avg. Collision (%): 0.22 -> 0.18 | Helped |
| Add E³AD to UniAD | UniAD · nuScenes open-loop | Avg. Collision (%): 0.31 -> 0.23* | Helped |
| Add E³AD to VAD-Tiny | VAD-Tiny · nuScenes open-loop | Avg. L2 (m): 0.78 -> 0.68 | Helped |
| Add E³AD to UniAD-Base | UniAD-Base · Bench2Drive | Driving score: 45.81 -> 50.07 | Helped |
| Add E³AD to VAD | VAD · Bench2Drive | Success rate (%): 15.00 -> 19.54 | Helped |
| Add EEG-video contrastive learning (ablation) | VAD-Base with frozen Driving-Thinking encoder · nuScenes open-loop | Avg. L2 (m): 0.70 -> 0.61 | Helped |
| Add EEG-video contrastive learning (ablation) | VAD-Base with frozen Driving-Thinking encoder · nuScenes open-loop | Avg. Collision (%): 0.25 -> 0.18 | Helped |
| Use expert-driver EEG rather than novice-driver EEG (ablation) | Driving-Thinking model · nuScenes open-loop | Avg. L2 (m): 0.67 -> 0.61 | Helped |
| Use expert and novice EEG together rather than expert EEG only (ablation) | Driving-Thinking model · nuScenes open-loop | Avg. Collision (%): 0.21 -> 0.18 | Helped |
| Interact with planning features instead of attaching cognition to spatio-temporal features (ablation) | E³AD (VAD-Base) · nuScenes open-loop | Avg. Collision (%): 0.22 -> 0.18 | Helped |
| Double the VAD-Base L2-loss coefficient (ablation) | VAD-Base · nuScenes open-loop | Avg. Collision (%): 0.22 -> 0.29 | Hurt |
| Increase sparse visual queries from 8 to 16 (ablation) | E³AD spatio-temporal feature attachment · nuScenes open-loop | Avg. Collision (%): 0.21 -> 0.27 | Hurt |
Surprising findings
- Unfreezing the Driving-Thinking model during stage-2 training removed the gains, which the authors attribute to forgetting of driving cognition.
- Video encoder without LaBraM contrastive alignment gave no improvement, indicating the gain comes from EEG alignment rather than extra visual capacity.
Limitations
- Paired EEG-video dataset is small (20 drivers, about 2,367 clips).
- Mechanisms by which EEG alignment helps planning are not fully explored.
- Paper states no explicit limitations section in the checklist; the authors promise to add one.
Experimental setup
Datasets: nuScenes Bench2Drive self-collected video-EEG driving dataset
Benchmarks: nuScenes Bench2Drive
Metrics: L2 error (m) collision rate (%) driving score success rate (%) FPS
Baselines: ST-P3 VAD-Base VAD-Tiny UniAD GenAD BEV-Planner LAW Ego-MLP
Builds on: LaBraM CLIP VAD UniAD Video Swin Transformer
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.87)
- Continuously Learning, Adapting, and Improving: A Dual-Process Approach to Autonomous Driving (NeurIPS 2024 · similarity 0.87)
- Enhancing End-to-End Autonomous Driving with Latent World Model (ICLR 2025 · similarity 0.86)
- SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.86)
- $AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning (ICLR 2026 · similarity 0.86)
- DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.