Research map / 3D vision / 3D generation
UniPAD: A Universal Pre-training Paradigm for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniPAD, a self-supervised pre-training paradigm that masks LiDAR or multi-view image inputs, lifts features into a 3D voxel volume, and reconstructs RGB and depth via SDF-based differentiable volume rendering, with memory-friendly ray sampling.
Problem
Self-supervised pre-training for 3D perception in autonomous driving; existing 3D pre-training methods (contrastive, MAE-based) adapt 2D ideas and give coarse or sample-selection-dependent supervision, and no unified pre-training works across LiDAR and camera backbones.
Approach
Use 3D volumetric differentiable neural rendering (NeuS-style SDF field conditioned on voxel features) as a unified pretext decoder for both LiDAR and multi-view camera backbones, with depth-aware ray sampling to cut memory cost.
Key results
- UniPAD improves UVTR-L, UVTR-C and UVTR-M by 9.1, 7.7 and 6.9 NDS on nuScenes val (Table 6).
- UniPAD reaches 73.2 NDS for LiDAR-camera 3D detection on nuScenes val, and 79.4 mIoU for segmentation on val with SpUNet baseline improved by 6.1 mIoU.
- Depth-aware ray sampling reaches 32.9 NDS vs 32.5 for random sampling and 31.9 for dilation sampling, with dilation using 1.4x memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply UniPAD pre-training | UVTR-L · nuScenes val, LiDAR 3D detection | NDS: 67.7 -> 70.6 | Helped |
| apply UniPAD pre-training | UVTR-C · nuScenes val, camera 3D detection | NDS: 45.0 -> 47.4 | Helped |
| apply UniPAD pre-training | UVTR-M · nuScenes val, camera-LiDAR 3D detection | NDS: 70.2 -> 73.2 | Helped |
| apply UniPAD pre-training | SpUNet · nuScenes Lidar-Seg val | mIoU: 73.3 -> 79.4 | Helped |
| vary the camera input mask ratio (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 31.9; 32.9; 32.3; 32.1; 31.4 | Mixed |
| vary SDF and RGB decoder depth (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 31.3; 31.9; 32.1; 32.9; 32.7 | Mixed |
| vary decoder width (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.1; 32.9; 32.5; 32.9; 32.4 | Mixed |
| replace NeuS rendering with UniSurf or VolSDF (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.9 -> 32.5; 32.9 -> 32.8 | Hurt |
| replace depth-aware ray sampling with random or dilation sampling (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.9 -> 32.5; 32.9 -> 31.9 | Hurt |
| remove feature projection during fine-tuning (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.9 -> 30.2 | Hurt |
| remove feature projection during pre-training (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.9 -> 31.1 | Hurt |
| share feature projection parameters between pre-training and fine-tuning (ablation) | UniPAD with UVTR-C · nuScenes val, camera detection | NDS: 32.9 -> 32.1 | Hurt |
Surprising findings
- Optimal masking ratio for camera pre-training is 0.3, lower than typical MAE ratios, which the authors attribute to the difficulty of rendering images from volume representations.
Limitations
- Point and image features must be explicitly transformed into volumetric representations, increasing memory usage as voxel resolution grows.
- Evaluated only on nuScenes.
Experimental setup
Datasets: nuScenes
Benchmarks: nuScenes detection val nuScenes LiDAR segmentation val nuScenes LiDAR segmentation test
Metrics: NDS mAP mIoU
Baselines: UVTR FCOS3D SpUNet PointContrast Occupancy-based (ALSO) MAE-based (GD-MAE) Contrast-based (pixel-to-point) BEVDet BEVDepth BEVFormer
Builds on: NeuS UVTR MAE ALSO UniSurf
Closest papers by meaning
- OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection (NeurIPS 2025 · similarity 0.88)
- VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving (CVPR 2025 · similarity 0.87)
- VLM2Scene: Self-Supervised Image-Text-LiDAR Learning with Foundation Models for Autonomous Driving Scene Understanding (AAAI 2024 · similarity 0.87)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.86)
- UniDrive: Towards Universal Driving Perception Across Camera Configurations (ICLR 2025 · similarity 0.86)
- UniScene: Unified Occupancy-centric Driving Scene Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.