Research map / 3D vision / 3D generation
AlignMiF: Geometry-Aligned Multimodal Implicit Field for LiDAR-Camera Joint Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes why jointly fusing LiDAR and camera in one implicit neural field hurts one modality, and proposes AlignMiF with decomposed hash encodings, geometry-aware alignment, and shared geometry initialization to improve both.
Problem
Multimodal implicit fields fusing LiDAR and camera data for novel view synthesis fail to improve both modalities at once; optimizing one often degrades the other, apparently due to sensor misalignment.
Approach
Decompose the hash encoding per modality, align only the coarse geometry levels across modalities via a geometry-aware alignment module (GAA), and initialize both modalities from a shared hash grid pretrained on LiDAR (SGI), so coarse geometry is shared while fine details stay modality-specific.
Key results
- On KITTI-360, AlignMiF reaches 25.31 image PSNR, +2.01 over UniSim-SF (tuned for camera) and 0.081 LiDAR Chamfer Distance, beating LiDAR-NeRF's 0.094.
- On Waymo, AlignMiF reaches 29.78 PSNR (+3.11 over UniSim-SF) and 13.8%/14.2% lower LiDAR Chamfer Distance than single-modality baselines across the two datasets.
- Ablation on KITTI-360 shows GAA (concatenation) and SGI each improve multimodal performance, and combining them gives 25.20 PSNR and 0.077 C-D.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine LiDAR and camera using AlignMiF rather than implicit fusion | LiDAR-camera novel view synthesis · KITTI-360 | PSNR: 23.30 (-) -> 25.31 (+) | Helped |
| combine LiDAR and camera using AlignMiF rather than implicit fusion | LiDAR-camera novel view synthesis · Waymo | PSNR: 26.67 (-) -> 29.78 (+) | Helped |
| use AlignMiF instead of single-modality LiDAR synthesis | LiDAR-NeRF · KITTI-360 | Chamfer Distance: 0.094 -> 0.081 (+) | Helped |
| use AlignMiF instead of single-modality LiDAR synthesis | LiDAR-NeRF · Waymo | Chamfer Distance: 0.197 -> 0.169 (+) | Helped |
| add Shared Geometry Initialization (SGI) without GAA (ablation) | AlignMiF · KITTI-360 | PSNR: 24.45 -> 24.70 (+) | Helped |
| add Geometry-Aware Alignment (GAA) without SGI (ablation) | AlignMiF · KITTI-360 | PSNR: 24.45 -> 24.64 (+) | Helped |
| add both SGI and GAA (ablation) | AlignMiF · KITTI-360 | PSNR: 24.45 -> 25.20 (+) | Helped |
| initialize using a loaded pre-trained LiDAR encoder (ablation) | AlignMiF SGI · KITTI-360 | PSNR: 24.45 -> 24.34 | Hurt |
| fix the pre-trained LiDAR encoder (ablation) | AlignMiF SGI · KITTI-360 | PSNR: 24.45 -> 18.60 | Hurt |
| detach RGB density gradients (ablation) | AlignMiF · KITTI-360 | PSNR: 24.45 -> 12.41 | Hurt |
| use concatenation rather than addition for GAA fusion (ablation) | AlignMiF GAA · KITTI-360 | Chamfer Distance: 0.081 -> 0.079 | Helped |
| share coarse geometry rather than align it (ablation) | AlignMiF GAA · KITTI-360 | PSNR: 24.45 -> 24.45 | No clear effect |
Surprising findings
- Naive implicit LiDAR-camera fusion often fails to beat single-modality models on real-world data, even though it helps on the synthetic AIODrive dataset.
- Simply fixing a pretrained LiDAR encoder or detaching camera density gradients degrades results badly (e.g., PSNR drops to 18.60 and 12.41 in ablations).
Limitations
- Modality fusion was studied on static scenes; dynamic foreground was not considered in the UniSim re-implementation.
- The fusion module choice (concatenation) is simple; more powerful fusion modules are left for future work.
- Improvements in camera PSNR are modest on some ablation variants (e.g., 24.70 vs 25.20 combined).
Experimental setup
Datasets: KITTI-360 Waymo Open Dataset AIODrive
Benchmarks: KITTI-360 Waymo Open Dataset AIODrive
Metrics: PSNR SSIM LPIPS Chamfer Distance F-score MAE
Baselines: i-NGP LiDAR-NeRF UniSim-SF StreetSurf
Builds on: NeRF instant-NGP UniSim LiDAR-NeRF Neuralangelo
Closest papers by meaning
- GeoNLF: Geometry guided Pose-Free Neural LiDAR Fields (NeurIPS 2024 · similarity 0.88)
- LiFi: LiDAR Generation from Multi-View Images via Geometric and Semantic Collaborative Guidance (NeurIPS 2026 · similarity 0.87)
- STGC-NeRF: Spatial-Temporal Geometric Consistency for LiDAR Neural Radiance Fields in Dynamic Scenes (AAAI 2025 · similarity 0.86)
- How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentation (ICML 2025 · similarity 0.86)
- LiDAR4D: Dynamic Neural Fields for Novel Space-time View LiDAR Synthesis (CVPR 2024 · similarity 0.86)
- PF3plat: Pose-Free Feed-Forward 3D Gaussian Splatting for Novel View Synthesis (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.