Research map / 3D vision / NeRF and radiance fields
LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds LidaRF on Nerfacto by fusing sparse-UNet Lidar features with hash grid features, adding occlusion-aware curriculum depth supervision from accumulated Lidar, and Lidar-projected augmented training views for street-scene NeRF novel view synthesis.
Problem
NeRF reconstruction and novel view synthesis on street scenes suffer from nearly collinear, sparse forward camera trajectories and low-texture road surfaces, and rendering from lane-shifted views is hard.
Approach
Fuse explicit Lidar point-cloud features (3D sparse UNet encoding queried by FRNN) with hash-grid radiance features, supervise depth from temporally accumulated Lidar with a curriculum occlusion-aware filter that adapts to NeRF's rendered depth, and augment training with Lidar-projected lateral-shifted views.
Key results
- On Pandaset interpolation, LidaRF reaches 27.255 PSNR, 0.812 SSIM and 0.224 LPIPS, beating Nerfacto (27.122, 0.804, 0.268) and UniSim (26.014).
- On 3.7m lane-shift extrapolation, FID drops to 126.0 versus 151.0 for Nerfacto and 141.3 for UniSim.
- On NuScenes, LidaRF reaches 31.162 PSNR and 0.211 LPIPS versus 29.377 PSNR and 0.349 LPIPS for S-NeRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LidaRF instead of Nerfacto | Pandaset novel-view synthesis · Pandaset interpolation | PSNR: 27.122 -> 27.255 | Helped |
| Use LidaRF instead of Nerfacto | Pandaset novel-view synthesis · Pandaset lane shift | FID @ 3.7m: 151.0 -> 126.0 | Helped |
| Use LidaRF instead of S-NeRF | NuScenes novel-view synthesis · NuScenes | PSNR: 29.377 -> 30.629 | Helped |
| Add single-frame Lidar depth supervision (ablation) | Nerfacto · Pandaset lane shift | FID @ 3.7m: 151.0 -> 138.2 | Helped |
| Add 10-adjacent-frame Lidar depth supervision (ablation) | Nerfacto · Pandaset lane shift | FID @ 3.7m: 151.0 -> 137.7 | Helped |
| Use HPR-filtered accumulated Lidar depth supervision (ablation) | Nerfacto · Pandaset lane shift | FID @ 3.7m: 151.0 -> 139.1 | Helped |
| Use robust supervision for accumulated Lidar depth (ablation) | Nerfacto · Pandaset lane shift | FID @ 3.7m: 151.0 -> 131.7 | Helped |
| Add robust accumulated-depth supervision (ablation) | Nerfacto · Pandaset ablation | PSNR: 27.122 -> 27.090 | No clear effect |
| Add Lidar encoding to robust depth supervision (ablation) | Nerfacto · Pandaset lane shift | FID @ 3.7m: 131.7 -> 128.7 | Helped |
| Add Lidar encoding to robust depth supervision (ablation) | Nerfacto · Pandaset interpolation | PSNR: 27.090 -> 27.219 | Helped |
| Add augmented-view supervision (ablation) | LidaRF with robust depth and Lidar encoding · Pandaset lane shift | FID @ 3.7m: 128.7 -> 126.0 | Helped |
| Add augmented-view supervision (ablation) | LidaRF with robust depth and Lidar encoding · Pandaset interpolation | PSNR: 27.219 -> 27.254 | Helped |
Surprising findings
- Naive single-frame or 10-frame Lidar depth supervision can reduce PSNR relative to Nerfacto, and offline hidden point removal lags far behind the adaptive occlusion scheme.
Limitations
- Handles only static background; dynamic objects are masked out and not modeled.
- Augmented-view supervision gives only modest quantitative gains on interpolation metrics.
Experimental setup
Datasets: Pandaset NuScenes Argoverse2
Benchmarks: Pandaset NuScenes Argoverse2
Metrics: PSNR SSIM LPIPS FID
Baselines: Instant-NGP Mip-NeRF 360 Nerfacto UniSim S-NeRF NeRF-LiDAR-cGAN
Builds on: Nerfacto Instant-NGP Mip-NeRF 360 Point-NeRF URF
Closest papers by meaning
- NeRF-LiDAR: Generating Realistic LiDAR Point Clouds with Neural Radiance Fields (AAAI 2024 · similarity 0.91)
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.90)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.90)
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes (AAAI 2025 · similarity 0.90)
- LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation (CVPR 2025 · similarity 0.88)
- NeuRAD: Neural Rendering for Autonomous Driving (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.