Research map / Probabilistic and scientific modeling / Molecules and proteins
Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Drive-OccWorld is a vision-centric 4D occupancy and flow forecasting world model with semantic/motion-conditional normalization, flexible action conditioning, and an occupancy-based planner for continuous forecasting and end-to-end planning.
Problem
Existing driving world models focus on video generation or pretraining rather than using future forecasting to improve end-to-end planning safety, and lack controllable action-conditioned 4D occupancy forecasting.
Approach
Adds semantic- and motion-conditional normalization to a BEV memory queue, an action-conditioned autoregressive transformer decoder predicting future semantic occupancy and backward centripetal flow, and an occupancy-cost-based planner that feeds predicted trajectories back as action conditions for continuous rollout.
Key results
- On nuScenes inflated GMO forecasting, Drive-OccWorld reports a 9.4% gain in weighted mIoU over Cam4DOcc and a 5.1% gain in VPQ_f over prior SoTA.
- On nuScenes end-to-end planning, Drive-OccWorld^P achieves avg L2 0.85 m and collision 0.29% under the NoAvg protocol vs UniAD 1.03 m / 0.31%.
- Ablations show ego-motion conditional normalization gives +1.9 mIoU_f and cross-attention action conditioning outperforms addition.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic- and motion-conditional normalization in memory (ego-motion) (ablation) | Drive-OccWorld memory queue · nuScenes Inflated GMO ablation | mIoU_f (1s): +1.9 (26.4 to 28.3) | Helped |
| add agent-motion conditional normalization (ablation) | Drive-OccWorld memory queue · nuScenes Inflated GMO ablation | VPQ_f*: +0.9 (33.5 to 34.4) | Helped |
| add semantic conditional normalization (ablation) | Drive-OccWorld memory queue · nuScenes Inflated GMO ablation | mIoU_f (1s): +0.2 (26.4 to 26.6) | Helped |
| add cross-attention action conditioning instead of addition to BEV queries (ablation) | Drive-OccWorld world decoder · nuScenes Inflated GMO ablation | mIoU_f (1s): 27.1 vs 27.4 addition (Fourier); cross-attention more effective per text | Helped |
| add trajectory action condition (ablation) | Drive-OccWorld world decoder · nuScenes Inflated GMO controllability | tilde mIoU_f: +1.0 (26.8 to 27.8) | Helped |
| add command action condition (ablation) | Drive-OccWorld world decoder · nuScenes Inflated GMO controllability | mIoU_c: +0.5 (28.7 to 29.2) | Helped |
| remove agent-safety cost from occupancy cost (ablation) | Drive-OccWorld planner · nuScenes planning | collision avg (%): 0.16 without vs 0.09 full | Hurt |
| remove BEV refinement from planner (ablation) | Drive-OccWorld planner · nuScenes planning | L2 avg (m): 0.37 without vs 0.28 full | Hurt |
| use predicted trajectories as action conditions instead of GT actions | Drive-OccWorld · nuScenes Inflated GMO forecasting | mIoU_f (2s): 36.3 vs 36.3 | No clear effect |
| use GT trajectory actions instead of predicted for planning | Drive-OccWorld planner · nuScenes planning upper bound | L2 avg (m): 0.56 vs 0.85 | Helped |
Surprising findings
- Using predicted trajectories as action conditions during training and testing slightly outperforms using ground-truth actions for occupancy forecasting.
- Low-level action conditions (trajectory, velocity) help future forecasting most, while high-level commands mainly help current-frame mIoU.
Limitations
- Latency is dominated by the BEV encoder, limiting real-time use with more frames.
- Planning evaluated only open-loop on nuScenes with L2 and collision metrics.
- Ground-truth-action planning upper bound yields L2 of 0.56 avg, showing gap from predicted trajectories (0.85).
Experimental setup
Datasets: nuScenes nuScenes-Occupancy Lyft-Level5
Benchmarks: nuScenes nuScenes-Occupancy Lyft-Level5
Metrics: mIoU VPQ L2 error collision rate
Baselines: SPC OpenOccupancy-C PowerBEV-3D Cam4DOcc CONet-C ST-P3 UniAD VAD-Base OccNet Drive-WM BEV-Planner NMP
Builds on: BEVFormer Cam4DOcc ST-P3 OccWorld VAD
Closest papers by meaning
- COME: Adding Scene-Centric Forecasting Control to Occupancy World Model (NeurIPS 2025 · similarity 0.92)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.92)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.91)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.91)
- Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications (CVPR 2024 · similarity 0.90)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.