Research map / Point clouds / Segmentation and detection
NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NeRF-LiDAR, a NeRF-based LiDAR simulator that learns scene geometry, LiDAR point generation, raydrop, and semantic labels from real nuScenes images and LiDAR, and shows that simulated data can replace or pre-train for real-label segmentation.
Problem
Labeling LiDAR point clouds for autonomous driving is expensive, and existing LiDAR simulators (virtual-world or mesh-reconstruction based) have domain gaps and require dense scans or manual 3D annotation.
Approach
Learn an unbounded, hash-encoded NeRF of driving scenes from multi-view images, sparse LiDAR depth, and weak 2D semantic labels; render LiDAR points and labels by simulating a virtual LiDAR; and refine them with a learned U-Net raydrop module supervised by point-wise and feature-level (VGG or segmentation-network) alignment losses.
Key results
- Sim20k + real1k reaches 89.1 mIoU on Test Set 1 versus 88.4 for real 10k and 86.5 for real 1k, with simulation-only training at 80.3 mIoU.
- On LiDARSim comparison (10k frames, 25 sequences), NeRF-LiDAR achieves 78.4 mIoU versus 57.8 for LiDARSim.
- Pre-training on simulated data raises 3D vehicle detection AP from 41.0 (real 1k) to 44.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune on real1k after training on Sim20k | Cylinder3D trained on NeRF-LiDAR · Test Set 1 | mIoU: 80.3 -> 89.1 | Helped |
| Add NeRF-LiDAR simulated data to Real 1k | Cylinder3D · Test Set 1, sim + real | mIoU: 86.5 -> 89.1 | Helped |
| Use NeRF-LiDAR synthetic data for pre-training, then fine-tune on real1k | CenterPoint · 3D detection | Average Precision: 41.0 to 44.0 | Helped |
| Add additional vehicles through object placement (ablation) | CenterPoint trained on NeRF-LiDAR · 3D vehicle detection | Average Precision: 25.7 -> 28.4 | Helped |
| Replace no raydrop with random raydrop (ablation) | NeRF-LiDAR point-cloud generation · Test Set 2 | mIoU: 65.7 -> 63.5 | Hurt |
| Use learned raydrop instead of no raydrop (ablation) | NeRF-LiDAR point-cloud generation · Test Set 2 | mIoU: 65.7 -> 66.3 | Helped |
| Add VGG feature-level alignment (ablation) | NeRF-LiDAR with learned raydrop · Test Set 2 | mIoU: 66.3 -> 66.5 | Helped |
| Use RangeNet++ rather than VGG as feature extractor (ablation) | NeRF-LiDAR with learned raydrop · Test Set 2 | mIoU: 66.3 -> 69.9 | Helped |
| Add 20% manually labeled LiDAR frames to weak image labels (ablation) | NeRF-LiDAR label generation · Generated-label quality | mIoU: 84.45 -> 87.07 | Helped |
| Increase manually labeled LiDAR frames from 20% to 50% (ablation) | NeRF-LiDAR label generation · Generated-label quality | mIoU: 87.07 -> 89.66 | Helped |
| Remove LiDAR depth supervision (ablation) | NeRF-LiDAR reconstruction and generation · Test Set 1 | mIoU: 84.4 -> 63.7 | Hurt |
| Increase the RGB balance weight above 1.0 (ablation) | NeRF-LiDAR reconstruction · Test Set 1 | mIoU: 81.5 -> 77.7 | Hurt |
Surprising findings
- Simulated-only training at 80.3 mIoU is close to training on 1k real labeled frames (86.5), while adding 1k real frames to simulation outperforms 10k real frames.
- Random raydrop reduced mIoU (65.7 to 63.5) relative to no raydrop, while learned raydrop improved it.
Limitations
- Only 30 nuScenes sequences (~4km) reconstructed due to compute limits, and LiDARSim comparison reproduced by the authors since official code is unavailable.
- Requires labeled LiDAR frames for best label quality; relies on weak SegFormer image labels that may contain outliers.
- Dynamic objects and object placement are handled by separate, lightly described procedures.
Experimental setup
Datasets: nuScenes Cityscapes Mapillary BDD IDD
Benchmarks: nuScenes LiDAR segmentation (Test Set 1) nuScenes validation set (Test Set 2) nuScenes 3D detection (CenterPoint)
Metrics: mIoU AP MAE Recall@50
Baselines: CARLA LiDARSim Urban-NeRF Panoptic Neural Fields LiDARGen
Builds on: NeRF Unbounded NeRF Instant-NGP hash encoding Semantic NeRF Cylinder3D RangeNet++
Closest papers by meaning
- LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes (CVPR 2024 · similarity 0.91)
- NeuRAD: Neural Rendering for Autonomous Driving (CVPR 2024 · similarity 0.89)
- GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting (ICLR 2025 · similarity 0.89)
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes (AAAI 2025 · similarity 0.89)
- LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis (CVPR 2024 · similarity 0.88)
- LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.