Research map / Reinforcement learning / Visual and representation RL
From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Policy World Model (PWM), a unified autoregressive Transformer (initialized from Show-o) that pretrains on action-free video forecasting with a compact 28-token-per-frame tokenizer and dynamic focal loss, then predicts text, future frames, and actions for front-camera driving planning.
Problem
World models for autonomous driving are typically decoupled from trajectory planning, and unified world-model-plus-planning approaches do not explicitly use the learned world knowledge to improve planning.
Approach
Pretrain on action-free video generation, then fine-tune to forecast text and future frames before predicting actions, so planning is conditioned on forecasted future states (collaborative state-action prediction), with parallel per-frame token generation via a context-guided two-branch tokenizer and a dynamic focal loss that upweights temporally changing tokens.
Key results
- On nuScenes validation without ego status, PWM reaches 0.07% average collision rate, versus 0.11% for Drive-OccWorld.
- On NAVSIM NavTest with front camera only, PWM achieves PDMS 88.1, matching DiffusionDrive (camera+LiDAR, 88.1) and exceeding DrivingGPT (82.4).
- Ablation shows video pretraining plus both dynamic focal losses lowers nuScenes average L2 from 3.34 m to 0.78 m and collision from 1.51% to 0.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add action-free video pre-training (ablation) | PWM · nuScenes planning | Avg. L2 (m): 3.34 -> 2.29 | Helped |
| add action-free video pre-training (ablation) | PWM · NAVSIM planning | PDMS: 77.8 -> 80.7 | Helped |
| add Dynamic Focal Loss during fine-tuning (ablation) | pre-trained PWM · nuScenes planning | Avg. collision (%): 1.05 -> 0.56 | Helped |
| add Dynamic Focal Loss during pre-training (ablation) | pre-trained PWM · NAVSIM planning | PDMS: 80.7 -> 86.3 | Helped |
| apply Dynamic Focal Loss during both pre-training and fine-tuning (ablation) | PWM · NAVSIM planning | PDMS: 77.8 -> 88.1 | Helped |
| increase video pre-training data from 0% to 100% (ablation) | PWM with 10-frame forecasts · nuScenes planning | Avg. L2 (m): 2.95 -> 0.78 | Helped |
| add 10-frame future forecasting during training (ablation) | PWM with 100% video pre-training · nuScenes planning | Avg. collision (%): 0.13 -> 0.07 | Helped |
| add 10-frame future forecasting during training (ablation) | PWM · NAVSIM planning | EP: 82.4 -> 81.8 | Hurt |
| increase forecast horizon from 10 to 15 frames (ablation) | PWM · NAVSIM planning | PDMS: 88.1 -> 88.0 | Hurt |
| add scene-description or action-description generation (ablation) | PWM · nuScenes validation | Avg. L2 (m): 0.77 -> 0.77; 0.77 -> 0.78 | No clear effect |
| set DFL β to 0.4 rather than 0.1 (ablation) | PWM · nuScenes validation | Avg. L2 (m): 0.82 -> 0.78 | Helped |
Surprising findings
- Without video pretraining, adding future-frame prediction during fine-tuning harms planning relative to directly predicting waypoints.
- Adding scene- or action-level text generation gave no significant improvement in forecasting or planning.
Experimental setup
Datasets: OpenDV-YouTube nuScenes NAVSIM CC12M FineWeb
Benchmarks: nuScenes NAVSIM
Metrics: L2 error collision rate PDMS FVD LPIPS PSNR
Baselines: ST-P3 UniAD VAD-Base BEV-Planner Omni-Q LAW Drive-OccWorld DiffusionDrive DrivingGPT TransFuser
Builds on: Show-o VQ-GAN DrivingGPT
Closest papers by meaning
- DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning (NeurIPS 2026 · similarity 0.89)
- Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (CVPR 2024 · similarity 0.89)
- World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry (NeurIPS 2026 · similarity 0.89)
- AdaWorld: Learning Adaptable World Models with Latent Actions (ICML 2025 · similarity 0.88)
- PWM: Policy Learning with Multi-Task World Models (ICLR 2025 · similarity 0.88)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.