Research map / Image generation / Autoregressive and latent generation
Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Copilot4D tokenizes lidar point clouds with a VQVAE (depth-rendering decoder) and forecasts future tokens with a MaskGIT-derived absorbing-uniform discrete diffusion spatio-temporal Transformer world model.
Problem
Scaling unsupervised world models for autonomous driving lags behind GPT-style language modeling because lidar observations are complex and unstructured and decoding tens of thousands of tokens per frame needs a scalable generative model. The paper targets point cloud forecasting on NuScenes, KITTI
Approach
Tokenize each lidar frame into BEV discrete tokens with a VQVAE whose decoder uses implicit occupancy and differentiable depth rendering, then predict future tokens with discrete diffusion. The discrete diffusion is a MaskGIT recast with uniform noise injected into unmasked tokens and loss applied to all positions, enabling resampling, plus classifier-free guidance on past context.
Key results
- On NuScenes 1s prediction, Copilot4D reaches Chamfer 0.36 versus 1.41 for 4D-Occ (about 75% reduction).
- On Argoverse2 3s prediction, Chamfer falls from 1.99 (4D-Occ) to 0.55 for Copilot4D.
- Classifier-free guidance with w=1.0 reduces NuScenes 3s Chamfer by 60% (1.40 to 0.56); the proposed discrete diffusion reduces MaskGIT Chamfer 0.82 to 0.58 (29%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · NuScenes 1s | Chamfer (within ROI): 1.41 -> 0.36 | Helped |
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · NuScenes 3s | Chamfer (within ROI): 1.40 -> 0.58 | Helped |
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · KITTI Odometry 1s | Chamfer (within ROI): 0.51 -> 0.18 | Helped |
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · KITTI Odometry 3s | Chamfer (within ROI): 0.96 -> 0.45 | Helped |
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · Argoverse 2 Lidar validation 1s | Chamfer (within ROI): 1.42 -> 0.26 | Helped |
| Use Copilot4D instead of 4D-Occ | point-cloud forecasting · Argoverse 2 Lidar validation 3s | Chamfer (within ROI): 1.99 -> 0.55 | Helped |
| Add classifier-free guidance (w=1.0) (ablation) | Copilot4D · NuScenes 3s | Chamfer (within ROI): 1.40 -> 0.56; 60%↓ | Helped |
| Set classifier-free guidance weight to 1.0 rather than 2.0 (ablation) | Copilot4D · NuScenes 3s | L1 Med: 0.14 -> 0.13 | No clear effect |
| Replace MaskGIT with the proposed discrete diffusion changes (ablation) | world-model prediction · NuScenes 3s | Chamfer (within ROI): 0.82 -> 0.58; 29%↓ | Helped |
| Add spatial skipping to the tokenizer (ablation) | Copilot4D tokenizer · NuScenes reconstruction | Chamfer (within ROI): 0.148 -> 0.082 | Helped |
| Add spatial skipping to the tokenizer (ablation) | Copilot4D tokenizer · NuScenes reconstruction | L1 Median: 0.044 -> 0.044 | No clear effect |
| Use differentiable depth rendering with implicit representation rather than voxel-presence-only reconstruction (ablation) | Copilot4D tokenizer · point-cloud reconstruction | reconstruction fidelity | Helped |
Surprising findings
- The tokenizer (13M) and world model (39M) are smaller than a ResNet-34 or ResNet-101, yet outperform prior SOTA.
Limitations
- Evaluated only on point cloud forecasting with lidar; generalization to other robotic domains is not demonstrated.
- Requires separate models for 1s and 3s prediction, with per-dataset frame configurations.
Experimental setup
Datasets: NuScenes KITTI Odometry Argoverse2 Lidar
Benchmarks: NuScenes KITTI Odometry Argoverse2 Lidar
Metrics: Chamfer distance L1 Med L1 Mean AbsRel Med AbsRel Mean
Baselines: 4D Occupancy SPFNet S2Net ST3DCNN MaskGIT
Builds on: MaskGIT D3PM VQVAE Swin Transformer GPT-2
Closest papers by meaning
- MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction (CVPR 2025 · similarity 0.88)
- DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (CVPR 2025 · similarity 0.87)
- Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving (AAAI 2025 · similarity 0.87)
- Navigation World Models (CVPR 2025 · similarity 0.87)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.86)
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.