Research map / 3D vision / 3D generation
FreeVS: Generative View Synthesis on Free Driving Trajectory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FreeVS is a fully generative video-diffusion novel view synthesis method for driving scenes that conditions on colored LiDAR pseudo-images, enabling synthesis on free new trajectories without per-scene reconstruction.
Problem
Reconstruction-based novel view synthesis for driving scenes only renders well along the recorded ego trajectory; views off that trajectory have untrained rays and degrade severely, and no ground-truth images exist for novel trajectories.
Approach
Represent all view priors (color, geometry, camera pose) as pseudo-images obtained by projecting colored LiDAR point clouds into the target camera, use them as conditions for a video diffusion model that only performs image completion, and simulate viewpoint translation by pairing targets with mismatched neighboring-frame LiDAR priors during training.
Key results
- On novel camera synthesis (front-side views), FreeVS reaches SSIM 0.628, PSNR 20.70, LPIPS 0.283, beating the best baseline EmerNeRF (0.603 / 19.61 / 0.330).
- On novel trajectories at lateral offset ±4.0 m, FreeVS obtains FID 22.08 versus 80.44 for Street Gaussians, and mAP^LET 0.612 versus 0.544.
- On multi-view recorded-trajectory novel frame synthesis, FreeVS gets SSIM 0.730 / PSNR 24.96 / LPIPS 0.179, above Street Gaussians (0.702 / 22.47 / 0.314), though its front-view PSNR (25.30) trails Street Gaussians (30.80).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FreeVS instead of the strongest baseline, EmerNerf | NVS methods · Waymo Open Dataset novel camera synthesis, front-side cameras | SSIM: 0.603 -> 0.628 | Helped |
| use FreeVS instead of the strongest baseline, StreetGaussian | NVS methods · Waymo Open Dataset novel trajectories, y±0.0 m | mAP LET: 0.826 -> 0.816 | Hurt |
| use FreeVS instead of StreetGaussian | NVS methods · Waymo Open Dataset novel trajectories, y±1.0 m | mAP LET: 0.738 -> 0.786 | Helped |
| use FreeVS instead of StreetGaussian | NVS methods · Waymo Open Dataset novel trajectories, y±2.0 m | mAP LET: 0.682 -> 0.724 | Helped |
| use FreeVS instead of StreetGaussian | NVS methods · Waymo Open Dataset novel trajectories, y±4.0 m | mAP LET: 0.544 -> 0.612 | Helped |
| use FreeVS instead of StreetGaussian | NVS methods · Waymo Open Dataset recorded trajectories, front view | SSIM: 0.903 -> 0.787 | Hurt |
| use FreeVS instead of the strongest baseline | NVS methods · Waymo Open Dataset recorded trajectories, multi-view | SSIM: 0.702 -> 0.730 | Helped |
| remove color from pseudo-image view priors (ablation) | FreeVS · Waymo Open Dataset view-prior ablation, multi-view | SSIM: 0.704 -> 0.701 | Hurt |
| remove LiDAR inputs (ablation) | FreeVS · Waymo Open Dataset view-prior ablation, y±2.0m | mAP LET: 0.690 -> 0.013 | Hurt |
| use separate raw priors instead of unified pseudo-image projection (ablation) | FreeVS · Waymo Open Dataset view-prior ablation, y±2.0m | mAP LET: 0.690 -> 0.028 | Hurt |
| replace 2D-Conv encoder with 2D-Attn encoder (ablation) | FreeVS · Waymo Open Dataset view-prior ablation, multi-view | SSIM: 0.704 -> 0.706 | No clear effect |
| sample target frames within ±2 frames (ablation) | FreeVS viewpoint transformation simulation · Waymo Open Dataset novel trajectories, y±2.0m | mAP LET: 0.704 -> 0.710 | Helped |
Surprising findings
- Feeding raw reference images plus camera pose transformation matrices to the diffusion model led it to ignore the pose condition and move the camera in familiar directions, which pseudo-image conditioning avoided.
- On the recorded front-view benchmark, FreeVS's PSNR (25.30) is lower than Street Gaussians (30.80), so the generative method is not uniformly best on recorded trajectories.
Limitations
- Relies on LiDAR point inputs, so it cannot operate without LiDAR data.
- Evaluation of novel trajectories depends on an off-the-shelf 3D detector as a proxy for geometric fidelity rather than ground truth.
- Front-view PSNR and LPIPS on recorded trajectories are weaker than Street Gaussians in the single-camera setting.
Experimental setup
Datasets: Waymo Open Dataset
Benchmarks: Waymo Open Dataset novel camera synthesis Waymo Open Dataset novel trajectory synthesis Waymo Open Dataset novel frame synthesis
Metrics: SSIM PSNR LPIPS FID LET-mAP
Baselines: 3D-GS EmerNeRF Street Gaussians
Builds on: Stable Video Diffusion 3D Gaussian Splatting MV-FCOS3D++
Closest papers by meaning
- FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes (CVPR 2025 · similarity 0.92)
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- Free3D: Consistent Novel View Synthesis without 3D Representation (CVPR 2024 · similarity 0.87)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.86)
- Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene (CVPR 2025 · similarity 0.86)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.