Research map / Label-efficient and robust learning / Recognition and long-tail
Enhancing End-to-End Autonomous Driving with Latent World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LAW, a self-supervised latent world model that predicts future scene latents from current latents and predicted ego waypoints, added to perception-free and perception-based end-to-end driving planners.
Problem
Improving scene feature representations for end-to-end autonomous driving planners without extra perception annotations, using temporal self-supervision that accounts for ego actions.
Approach
Predict future visual latents (perspective-view or BEV) from current latents concatenated with predicted ego waypoints via transformer blocks, supervised by MSE to latents of the future frame, avoiding costly image-generation diffusion world models.
Key results
- On nuScenes, the perception-based LAW reaches 0.49 m average L2 and 0.19% average collision versus VAD's 0.72 m and 0.22%.
- On NAVSIM test, LAW (perception-free) achieves 84.6 PDMS, above TransFuser (84.0) and PARA-Drive (84.0).
- On CARLA Town05 Long, LAW achieves 70.1 driving score, above Interfuser (68.3) and DriveAdapter (65.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add latent future prediction using visual latents and predicted trajectory (ablation) | LAW perception-free framework · nuScenes | L2 Avg. (m): 0.71 -> 0.61 | Helped |
| add predicted trajectory as an input to latent prediction (ablation) | LAW perception-free framework with visual latent input · nuScenes | L2 Avg. (m): 0.68 -> 0.61 | Helped |
| add latent future prediction using visual latents and predicted trajectory (ablation) | LAW perception-based framework · nuScenes | L2 Avg. (m): 0.54 -> 0.49 | Helped |
| add predicted trajectory as an input to latent prediction (ablation) | LAW perception-based framework with visual latent input · nuScenes | L2 Avg. (m): 0.52 -> 0.49 | Helped |
| add latent prediction (ablation) | LAW · NAVSIM | PDMS: 77.5 -> 84.6 | Helped |
| add latent prediction (ablation) | LAW · CARLA Town05 Long | DS: 67.9±2.1 -> 70.1±2.6 | Helped |
| predict future latents at 1.5s instead of 0.5s (ablation) | LAW · nuScenes | L2 Avg. (m): 0.61 -> 0.58 | Helped |
| extend latent prediction horizon from 1.5s to 3.0s (ablation) | LAW · nuScenes | L2 Avg. (m): 0.58 -> 0.63 | Hurt |
| extend latent prediction horizon to 10.0s (ablation) | LAW · nuScenes | L2 Avg. (m): 0.61 -> 0.72 | Hurt |
| use transformer blocks rather than a single-layer linear projection for the latent world model (ablation) | LAW · nuScenes | L2 Avg. (m): 0.70 -> 0.61 | Helped |
| use transformer blocks rather than a two-layer MLP for the latent world model (ablation) | LAW · nuScenes | L2 Avg. (m): 0.64 -> 0.61 | Helped |
| add autoregressive prediction of the 3s future latent after predicting 1.5s (ablation) | LAW front-view · nuScenes | L2 Avg. (m): 0.73 -> 0.69 | Helped |
Surprising findings
- Predicting latents 10 seconds into the future completely diminishes the gains of the world model.
- A single linear projection as the latent world model degrades performance well below the transformer-block default.
Limitations
- Gains on nuScenes perception-free setting are modest relative to perception-based; perception-free collision rate is worse than VAD's 0.22% average.
- CARLA IS is lower than several baselines (0.72 vs. 0.77 TransFuser for example)
- Multi-step and multi-frame extensions were only evaluated in appendix with front-view camera.
Experimental setup
Datasets: nuScenes NAVSIM OpenScene nuPlan CARLA
Benchmarks: nuScenes NAVSIM CARLA Town05 Long
Metrics: L2 displacement error collision rate PDMS NC DAC TTC Comfort EP route completion infraction score
Baselines: UniAD VAD BEV-Planner TransFuser PARA-Drive TCP ThinkTwice DriveAdapter Interfuser MILE Roach LAV
Builds on: PETR UniAD TCP VAD SOLOFusion
Closest papers by meaning
- Prioritizing Perception-Guided Self-Supervision: A New Paradigm for Causal Modeling in End-to-End Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- SEPT: Towards Efficient Scene Representation Learning for Motion Prediction (ICLR 2024 · similarity 0.87)
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.87)
- Navigation-Guided Sparse Scene Representation for End-to-End Autonomous Driving (ICLR 2025 · similarity 0.87)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.87)
- DualAD: Disentangling the Dynamic and Static World for End-to-End Driving (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.