Research map / 3D vision / Gaussian splatting
FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
FreeSim renders driving scenes from viewpoints several meters off the recorded trajectory by combining 3DGS (PVG) reconstruction with a diffusion image-enhancement model trained on degraded renderings, applied progressively during reconstruction.
Problem
Camera simulation for autonomous driving degrades badly from viewpoints beyond recorded ego trajectories because no ground-truth images exist for those off-trajectory views.
Approach
Reformulates novel-view generation as image enhancement of degraded renderings: trains a ControlNet-conditioned Stable Diffusion model on degraded extrapolated and Gaussian-perturbed renderings paired with real images, then progressively adds generated off-trajectory views during 3DGS reconstruction in small lateral steps.
Key results
- Off-trajectory FID at 3 m lateral shift: 18.6 for FreeSim vs 47.5 for PVG baseline, 47.6 StreetGS, 49.8 EmerNeRF, 81.3 S3Gaussian.
- On recorded views FreeSim PSNR 28.32 and SSIM 0.852, a slight drop from PVG's 29.19 and 0.866.
- Non-progressive expansion gives 29.7 FID at 3 m versus 18.6 for the default progressive setting with 0.5 m step size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the FreeSim generation-reconstruction method | PVG · WOD unrecorded views @3m | FID: 47.5 -> 18.6 | Helped |
| remove LiDAR condition (ablation implies adding it) (ablation) | FreeSim · WOD unrecorded views @3m | FID: 21.3 -> 18.6 | Helped |
| remove Gaussian perturbation (ablation implies adding it) (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.9 -> 18.6 | Helped |
| remove extrapolated rendering (ablation implies adding it) (ablation) | FreeSim · WOD unrecorded views @3m | FID: 20.3 -> 18.6 | Helped |
| set image-blending probability to 0.5 instead of the default 0.1 (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 20.4 | Hurt |
| disable image blending instead of using the default probability 0.1 (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 20.6 | Hurt |
| use a 0.25 m viewpoint expansion step instead of 0.5 m (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 19.5 | Hurt |
| use a 1.0 m viewpoint expansion step instead of 0.5 m (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 18.4 | Helped |
| use a 1.5 m viewpoint expansion step instead of 0.5 m (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 20.0 | Hurt |
| replace progressive reconstruction with non-progressive viewpoint expansion (ablation) | FreeSim · WOD unrecorded views @3m | FID: 18.6 -> 29.7 | Hurt |
Surprising findings
- Recorded-view PSNR drops slightly with the generative component, attributed to generation inconsistency, while off-trajectory FID improves dramatically.
Limitations
- Rolling shutter distortion and generative randomness cause blur in fine-detail regions such as grass; post-enhancement adds multi-step denoising cost.
- Training data is generated only with PVG, so the generator may not generalize well to NeRF-based renderings.
- Inaccurate depth causes ghosting and failure cases (e.g., window frames) at large deviations.
Experimental setup
Datasets: Waymo Open Dataset EmerNeRF candidate scenes (16 sequences)
Benchmarks: Waymo Open Dataset
Metrics: FID PSNR SSIM
Baselines: StreetGS EmerNeRF S3Gaussian PVG
Builds on: 3D Gaussian Splatting PVG ControlNet Stable Diffusion 1.5 FreeVS
Closest papers by meaning
- FreeVS: Generative View Synthesis on Free Driving Trajectory (ICLR 2025 · similarity 0.92)
- FRUC: Feedforward Dynamic Scene Reconstruction from Uncalibrated Collaborative Driving Views (NeurIPS 2026 · similarity 0.87)
- DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input (AAAI 2025 · similarity 0.87)
- Novel View Synthesis Under Large-Deviation Viewpoint for Autonomous Driving (AAAI 2025 · similarity 0.87)
- FlexDrive: Toward Trajectory Flexibility in Driving Scene Gaussian Splatting Reconstruction and Rendering (CVPR 2025 · similarity 0.87)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.