Research map / Point clouds / Point cloud classification
Visual Point Cloud Forecasting enables Scalable Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes visual point cloud forecasting (predicting future LiDAR-like point clouds from historical multi-view images) as a pre-training task, realized as ViDAR with a Latent Rendering operator, which improves BEVFormer, OccNet and UniAD across perception, prediction and planning on nuScenes.
Problem
Pre-training visual encoders for end-to-end autonomous driving that must jointly capture semantics, 3D geometry, and temporal dynamics, without requiring costly 3D annotations.
Approach
Forecast future point clouds from historical visual BEV features using an auto-regressive Future Decoder, with a Latent Rendering operator that replaces differentiable ray-casting to avoid ray-shaped features and learn discriminative 3D geometric latents.
Key results
- Point cloud forecasting: ViDAR reduces 1s-future Chamfer Distance from 1.88 (4D-Occ, LiDAR) to 1.25 m^2 using visual input only (~33% reduction).
- ViDAR-3D-det improves BEVFormer ResNet101 detection to 45.8% mAP / 54.8% NDS vs 41.5% / 51.7% for 3D-det pre-training.
- With ViDAR, UniAD average collision rate drops from 0.27% to 0.23% and avg L2 from 1.12 to 0.91 m, outperforming VAD on open-loop planning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply visual point cloud forecasting pre-training | BEVFormer on nuScenes 3D detection · nuScenes 3D detection, RN101 | mAP (%): 37.7 -> 42.6 | Helped |
| apply ViDAR pre-training | OccNet · nuScenes semantic occupancy prediction | mIoU (%): 24.35 -> 29.57 | Helped |
| apply ViDAR pre-training | UniAD · nuScenes motion forecasting | minADE (m): 0.75 -> 0.67 | Helped |
| apply ViDAR pre-training | UniAD · nuScenes planning, 3s average | Avg.Col. (3s) (%): 0.27 -> 0.23 | Helped |
| use Latent Rendering rather than no forecasting pre-training (ablation) | ViDAR for BEVFormer-small · nuScenes 3D detection | NDS (%): 44.11 -> 47.58 | Helped |
| replace Latent Rendering with differentiable ray-casting (ablation) | ViDAR for BEVFormer-small · nuScenes 3D detection | NDS (%): 44.11 -> 40.20 (-3.91 ) | Hurt |
| use 1 group of Latent Rendering instead of 16 groups (ablation) | ViDAR for BEVFormer-small · nuScenes 3D detection | NDS (%): 39.18 -> 47.58 | Hurt |
| fine-tune with half of the supervised samples (ablation) | ViDAR-pretrained BEVFormer · nuScenes 3D detection, half training subset | mAP: 39.4% mAP vs. 37.7% mAP | Helped |
Surprising findings
- Directly combining differentiable ray-casting with the encoder and decoder for visual point cloud forecasting pre-training decreases downstream detection NDS from 44.11% to 40.20%.
- ViDAR pre-trained with only 1/2 of the 3D annotations outperforms the fully supervised BEVFormer baseline (39.4% vs 37.7% mAP).
Limitations
- Pre-training limited to nuScenes Image-LiDAR sequences, a relatively small data scale.
- Auto-regressive prediction is constrained to a limited 1s input horizon in practice.
- Some reported planning averages are non-standard relative to the 3s timestamp metrics of prior work.
Experimental setup
Datasets: nuScenes COCO
Benchmarks: nuScenes
Metrics: Chamfer Distance mAP NDS mIoU IoU VPQ AMOTA minADE minFDE MR
Baselines: 4D-Occ UniAD BEVFormer OccNet ViP3D BEVerse VAD FCOS3D ImageNet-cls pre-training
Builds on: BEVFormer UniAD 4D-Occ Deformable DETR
Closest papers by meaning
- FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving (NeurIPS 2025 · similarity 0.88)
- UnO: Unsupervised Occupancy Fields for Perception and Forecasting (CVPR 2024 · similarity 0.87)
- Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving (ICLR 2025 · similarity 0.87)
- TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception (NeurIPS 2025 · similarity 0.86)
- SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving (CVPR 2026 · similarity 0.86)
- SEPT: Towards Efficient Scene Representation Learning for Motion Prediction (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.