Research map / Reinforcement learning / Visual and representation RL

Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)

NeurIPS 2025 · Reinforcement learning › Visual and representation RL · approach: Probabilistic and autonomous RL · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Raw2Drive is a dual-stream model-based RL method that trains a privileged world model and planner, then aligns a raw-sensor world model to it via rollout and head guidance, achieving state-of-the-art closed-loop driving on Bench2Drive/CARLA v2.

Problem

Applying reinforcement learning to end-to-end autonomous driving from raw sensor inputs is hard because raw sensor world models are high-dimensional and difficult to train, and prior MBRL work (Think2Drive) relied on privileged state inputs; prior IL methods also plateau on CARLA v2.

Approach

A dual-stream MBRL framework: a privileged (BEV mask) world model and policy are trained first, then a raw-sensor world model (BEVFormer encoder) is trained with rollout guidance (encoder, deterministic and stochastic state alignment, with single-sample stochastic states shared across both models) and the raw policy is fine-tuned using reward and continuation signals from the privileged model's he

Key results

• On Bench2Drive closed-loop benchmark, Raw2Drive reaches 71.36 Driving Score and 50.24% Success Rate, the best among raw-sensor end-to-end methods (DriveTransformer 63.46 DS / 35.01 SR).
• On CARLA Leaderboard 2.0 validation, Raw2Drive achieves 4.12 DS and 9.32 RC versus at most 0.92 DS for prior image-based IL methods.
• Raw2Drive uses 64 H800 GPU days in total, reduced to 40 GPU days when Think2Drive is reused.

Measured effects reported in the paper

Surprising findings

• Adding reward and continuation heads to the raw sensor world model degrades performance (DS drops from 83.5 to 34.5 on Dev10), so only the decoder head is used.
• Directly using the privileged policy on raw sensors gives DS 58.4, lower than after fine-tuning in the raw sensor world model (83.5).

Limitations

• Privileged input relies on ground-truth bounding boxes and HD-Map in training, which may need human annotation or perception in practice.
• Evaluation is limited to CARLA; long-route Leaderboard 2.0 scores are argued to be unreliable, and the method has not been validated in the real world.
• Raw sensor stream latency is dominated by the BEVFormer encoder (about 600 ms).

Experimental setup

Datasets: Bench2Drive CARLA Leaderboard 2.0 routes

Benchmarks: Bench2Drive CARLA Leaderboard 2.0 (validation and devtest)

Metrics: Driving Score Success Rate Route Completion Infraction Score Efficiency Comfort

Baselines: UniAD-Base VAD TCP-traj ThinkTwice DriveAdapter DriveTransformer AD-MLP Think2Drive

Builds on: DreamerV3 Think2Drive BEVFormer Roach

Closest papers by meaning

• RAD: Training an End-to-End Driving Policy via Large-Scale 3DGS-based Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• AdaWM: Adaptive World Model based Planning for Autonomous Driving (ICLR 2025 · similarity 0.87)
• Distill to Think, Foresee to Act: Cognitive-Physical Reinforcement Learning for Autonomous Driving (NeurIPS 2026 · similarity 0.86)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.86)
• CarPlanner: Consistent Auto-regressive Trajectory Planning for Large-Scale Reinforcement Learning in Autonomous Driving (CVPR 2025 · similarity 0.85)
• AlignDrive: Aligned Lateral-Longitudinal Planning for End-to-End Autonomous Driving (NeurIPS 2026 · similarity 0.85)