Research map / 3D vision / 3D generation
Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StreetNVS, a Wan-2.2-based video diffusion model that jointly conditions on sparse LiDAR reprojections, surround-view reference images, and camera poses via reference-enhanced ray-level relative camera attention, trained with a two-stage dense-to-sparse LiDAR curriculum for street-view novel-view synthesis.
Problem
Street-view novel-view synthesis degrades when target trajectories depart from recorded vehicle paths, because reprojected LiDAR becomes too sparse to guide video diffusion models.
Approach
Treat street-view NVS as multi-sensor fusion: combine sparse LiDAR reprojections (metric geometry), surround-view reference images (dense appearance), and camera poses inside a DiT via a reference-enhanced camera attention built on ray-level relative pose encodings (UCPE plus RoPE frame index), with a progressive dense-to-balanced LiDAR sparsity curriculum.
Key results
- At LiDAR sparsity 0.01 on Waymo, StreetNVS reaches PSNR 20.82, SSIM 0.619, LPIPS 0.311, FVD 9.12, beating the retrained StreetCrafter* (19.22 PSNR, FVD 12.29).
- Original StreetCrafter reaches 16.81 PSNR and FreeVS 13.19 PSNR under the same sparse setting.
- Ablation: removing the two-stage curriculum drops PSNR from 20.82 to 19.51; removing reference views drops it to 20.34.
Surprising findings
- Camera attention alone (no LiDAR) reaches only 17.60 PSNR, yet the strongest gains come from combining LiDAR with reference views and camera attention.
Limitations
- Evaluation limited to clips of moderate length; scaling to longer sequences is untested.
- Focused only on novel-view generation, not scene editing such as inserting or removing objects.
- Requires LiDAR and multi-camera rig data, limiting applicability to sensor-equipped vehicles.
Experimental setup
Datasets: Waymo Open Dataset
Benchmarks: Waymo Open Dataset (402 test clips)
Metrics: PSNR SSIM LPIPS FVD
Baselines: FreeVS Gen3C StreetCrafter WAN-2.1-VACE-14B StreetCrafter*
Builds on: StreetCrafter UCPE Wan 2.2 Flow Matching DiffSynth
Closest papers by meaning
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.92)
- LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes (CVPR 2024 · similarity 0.90)
- ViiNeuS: Volumetric Initialization for Implicit Neural Surface Reconstruction of Urban Scenes with Limited Image Overlap (CVPR 2025 · similarity 0.87)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.87)
- X-Drive: Cross-modality Consistent Multi-Sensor Data Synthesis for Driving Scenarios (ICLR 2025 · similarity 0.87)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.