Research map / Vision-language and multimodal / Multimodal LLMs

S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes S4-Driver, a self-supervised MLLM (PaLI) motion planner that lifts frozen perspective-view features into a sparse 3D spatio-temporal volume, trained on raw driving logs without human annotations.

Problem

Self-supervised end-to-end motion planning from camera images typically underperforms supervised multi-task methods, partly because MLLMs reason in 2D image space rather than the 3D space where vehicles plan, and because nuScenes is too small to train billion-parameter MLLMs.

Approach

Lift frozen multi-view, multi-frame PaLI visual features into a learned sparse 3D volume (gated by a learned gate with a learnable vacant feature), add relative 3D attention bias, and predict waypoints as text with meta-decisions and multi-sample averaging, all trained only on ego trajectories.

Key results

• On nuScenes, S4-Driver* (pretrained on WOMD-Planning-ADE) reaches average L2 0.31 m versus 0.37 m for VAD and 0.44 m for GPT-Driver, without human annotations.
• On WOMD-Planning-ADE, S4-Driver* achieves ADE@5s 0.655 and bADE@5s 0.830 versus 0.798 and 1.069 for vanilla PaLI.
• Using PaLI2-3B instead of PaLI3-5B gives ADE@5s 1.035 vs 0.693 on full WOMD-Planning-ADE, showing the gap widens with more data.

Measured effects reported in the paper

Surprising findings

• Camera inputs make a large difference on WOMD-Planning-ADE but not on nuScenes, unlike prior observations.
• Motion trajectory tokenization hurt performance compared with plain floating-number waypoints in text.

Limitations

• Open-loop evaluation only on 5s horizons; no closed-loop testing.
• WOMD-Planning-ADE uses in-house camera data and pretraining data that are not released.
• Requires billion-parameter MLLM with 128 TPU v4 for ~2.5 days of finetuning.

Experimental setup

Datasets: WOMD-Planning-ADE nuScenes in-house camera driving logs

Benchmarks: nuScenes WOMD-Planning-ADE

Metrics: L2 displacement error (1s, 2s, 3s) ADE@1s/3s/5s bADE (behavior-wise ADE)

Baselines: UniAD VAD PARA-Drive GPT-Driver DriveVLM OmniDrive DriveVLM (w/o CoT) MotionLM Vanilla PaLI PaLI2-3B

Builds on: PaLI SimpleBEV MotionLM nucleus sampling

Closest papers by meaning

• MAPLM: A Real-World Large-Scale Vision-Language Benchmark for Map and Traffic Scene Understanding (CVPR 2024 · similarity 0.90)
• DriveGPT4-V2: Harnessing Large Language Model Capabilities for Enhanced Closed-Loop Autonomous Driving (CVPR 2025 · similarity 0.89)
• STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving (NeurIPS 2025 · similarity 0.89)
• DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving (NeurIPS 2026 · similarity 0.89)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving (CVPR 2026 · similarity 0.89)
• Less Language, More Latents: Annotation-Efficient VLAs for Driving (NeurIPS 2026 · similarity 0.88)