Research map / 3D vision / 3D generation
Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vampire, which regulates dense intermediate 3D features in LSS-based multi-camera perception by supervising volume-rendered depth and semantic maps from camera and BEV views, evaluated on Occ3D-nuScenes occupancy, Panoptic nuScenes LiDAR segmentation, and nuScenes detection.
Problem
LSS-based multi-camera perception produces sparse 3D features unsuitable for dense tasks like 3D occupancy prediction, and naive densification via inpainting is uncontrolled and can generate features at wrong locations.
Approach
Treat occupancy as NeRF volume density and apply volume rendering to the dense 3D feature volume, supervising rendered depth and semantic maps (camera and BEV views) with LiDAR-projected labels so that the inpainting network is regulated at the feature level.
Key results
- On Occ3D-nuScenes, Vampire (ResNet-50, 256x704) reaches 28.3 mIoU, versus 27.8 for TPVFormer and 21.9 for OccFormer (R101), and just below CTF-Occ at 28.5.
- On Panoptic nuScenes LiDAR segmentation validation, Vampire reaches 66.4 mIoU with R50, above TPVFormer R50 at 59.3 and OccNet at 53.0, and below TPVFormer R101 at 68.9.
- On nuScenes detection without temporal fusion, Vampire achieves 0.301 mAP and 0.354 NDS, with mAP above BEVDet (0.286) but NDS below BEVDet (0.372).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sparse feature inpainting (ablation) | LSS-based Vampire · Occ3D-nuScenes | mIoU: 21.9 -> 23.8 | Helped |
| add sparse feature inpainting (ablation) | LSS-based Vampire · nuScenes LiDAR segmentation | mIoU: 56.5 -> 60.1 | Helped |
| add sparse feature inpainting (ablation) | LSS-based Vampire · nuScenes 3D object detection | NDS: 0.318 -> 0.316 | No clear effect |
| add depth consistency loss (ablation) | Vampire with inpainting · Occ3D-nuScenes | mIoU: 23.8 -> 24.9 | Helped |
| add depth consistency loss (ablation) | Vampire with inpainting · nuScenes LiDAR segmentation | mIoU: 60.1 -> 59.6 | Hurt |
| add depth consistency loss (ablation) | Vampire with inpainting · nuScenes 3D object detection | NDS: 0.316 -> 0.309 | Hurt |
| add semantic consistency loss (ablation) | Vampire with depth consistency loss · Occ3D-nuScenes | mIoU: 24.9 -> 25.8 | Helped |
| add semantic consistency loss (ablation) | Vampire with depth consistency loss · nuScenes LiDAR segmentation | mIoU: 59.6 -> 62.6 | Helped |
| add semantic consistency loss (ablation) | Vampire with depth consistency loss · nuScenes 3D object detection | NDS: 0.309 -> 0.318 | Helped |
| add camera-view regulation to BEV-view regulation (ablation) | Vampire · Occ3D-nuScenes | mIoU: 24.9 -> 25.8 | Helped |
| add BEV-view regulation to camera-view regulation (ablation) | Vampire · nuScenes LiDAR segmentation | mIoU: 61.9 -> 62.6 | Helped |
| add BEV-view regulation to camera-view regulation (ablation) | Vampire · nuScenes 3D object detection | NDS: 0.303 -> 0.318 | Helped |
Surprising findings
- Depth regulation alone degrades LiDAR segmentation and detection, and only the semantic loss recovers these drops.
- Vampire's mAVE (1.043) is much worse than the joint-training OccNet-style baselines (0.541 for BEVNet), which the authors attribute to missing occupancy flow annotations.
Limitations
- Supervision is sparse, derived from LiDAR point projections, which limits regulation quality.
- Detection velocity (mAVE) is poor, and NDS trails BEVDepth and BEVDet, attributed to negative transfer in multi-task training.
- Only single-frame supervision from six cameras and BEV is used; temporal supervision is not explored.
Experimental setup
Datasets: nuScenes Occ3D-nuScenes Panoptic nuScenes
Benchmarks: Occ3D-nuScenes Panoptic nuScenes nuScenes
Metrics: mIoU mAP NDS mAVE
Baselines: MonoScene BEVDet BEVFormer TPVFormer OccFormer UniOcc CTF-Occ OccNet VoxNet BEVDepth
Builds on: Lift-Splat-Shoot NeRF BEVDepth OccNet
Closest papers by meaning
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.86)
- UniPAD: A Universal Pre-training Paradigm for Autonomous Driving (CVPR 2024 · similarity 0.85)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.85)
- UniDrive: Towards Universal Driving Perception Across Camera Configurations (ICLR 2025 · similarity 0.84)
- VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving (CVPR 2025 · similarity 0.84)
- VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.