Research map / 3D vision / NeRF and radiance fields
SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
SOAC performs targetless, self-supervised spatio-temporal calibration of cameras and LiDAR for driving by training one NeRF per camera and alternately optimizing extrinsics/time offsets against the other NeRFs, restricted to overlapping visible regions via visibility grids.
Problem
Multi-sensor (camera and LiDAR) extrinsic and temporal calibration in unconstrained outdoor driving scenes without targets, where a single shared NeRF can overfit to single-sensor regions and get stuck in local minima.
Approach
Represent each camera with its own NeRF, calibrate each sensor by registering its observations against the NeRFs of the other sensors in a round-robin fashion, and mask rays with per-NeRF visibility grids so only overlapping regions drive the calibration; LiDAR trains all NeRFs and is calibrated against them, with NeRF delaying and bounded corrections for stability.
Key results
- On KITTI-360, SOAC reaches overall errors of 0.21°, 5.24 cm and 3.95 ms, versus about 2.24°, 56.34 cm and 27.07 ms for MOISST under the same setup.
- On LiDAR/camera calibration, SOAC gets 0.3° / 7.8 cm on KITTI-360 and 1.3° / 29.4 cm on Pandaset, versus MOISST 0.2° / 10.0 cm and 2.8° / 56.4 cm on Pandaset.
- Temporal and spatial calibration become ambiguous for straight, constant-speed driving, where translation/time-offset are wrong even though pose error is low.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SOAC instead of MOISST for LiDAR-camera calibration | LiDAR-camera calibration · KITTI-360 | translation error (cm): 10.0 ± 9.8 -> 7.8 ± 3.5 | Helped |
| Use SOAC instead of MOISST for LiDAR-camera calibration | LiDAR-camera calibration · KITTI-360 | rotation error (°): 0.2 ± 0.1 -> 0.3 ± 0.2 | Hurt |
| Use SOAC instead of MOISST for LiDAR-camera calibration | LiDAR-camera calibration · Pandaset | rotation error (°): 2.8 ± 2.3 -> 1.3 ± 0.8 | Helped |
| Use SOAC instead of MOISST for LiDAR-camera calibration | LiDAR-camera calibration · Pandaset | translation error (cm): 56.4 ± 17.2 -> 29.4 ± 13.6 | Helped |
| Use SOAC instead of MOISST | spatio-temporal sensor calibration · KITTI-360 | rotation error (°): 2.24° -> 0.21° | Helped |
| Use SOAC instead of MOISST | spatio-temporal sensor calibration · KITTI-360 | translation error (cm): 56.34 cm -> 5.24 cm | Helped |
| Use SOAC instead of MOISST | spatio-temporal sensor calibration · KITTI-360 | time offset error (ms): 27.07 ms -> 3.95 ms | Helped |
| Add semantic filtering of dynamic classes (ablation) | SOAC · Pandaset dynamic scenes | rotation error (°): 2.36° -> 0.41° | Helped |
| Add semantic filtering of dynamic classes (ablation) | SOAC · Pandaset dynamic scenes | translation error (cm): 79.17 cm -> 7.79 cm | Helped |
| Remove NeRF delaying (ablation) | SOAC · nuScenes, 5-camera calibration | calibration error | Hurt |
Surprising findings
- On straight-line constant-speed driving, spatial and temporal parameters cannot be disentangled, yielding low pose error but incorrect extrinsic and time offset.
- On LiDAR/camera calibration, SOAC is comparable or worse than MOISST on KITTI-360 rotation (0.3° vs 0.2°).
Limitations
- Training time grows with the number of cameras since one NeRF is trained per camera (about 8 minutes per epoch with 5 cameras on nuScenes).
- Requires a reference sensor with known trajectory and scenes with nearby structures for precise calibration.
- Dynamic objects must be removed via semantic filtering; temporal calibration is unreliable on constant-speed straight trajectories.
Experimental setup
Datasets: KITTI-360 nuScenes Pandaset
Benchmarks: KITTI-360 nuScenes Pandaset
Metrics: rotation error (degrees) translation error (cm) temporal error (ms)
Baselines: MOISST LCCNet Pandey et al. feature-based calibration INF AsyncNeRF
Builds on: MOISST NeRF NeRF2NeRF BARF
Closest papers by meaning
- UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving (ICLR 2024 · similarity 0.88)
- Multi-Level Neural Scene Graphs for Dynamic Urban Environments (CVPR 2024 · similarity 0.85)
- LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes (CVPR 2024 · similarity 0.85)
- NeuRAD: Neural Rendering for Autonomous Driving (CVPR 2024 · similarity 0.84)
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes (AAAI 2025 · similarity 0.84)
- Swift-Mapping: Online Neural Implicit Dense Mapping in Urban Scenes (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.