Research map / 3D vision / 3D generation
ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReconDreamer, which fine-tunes a driving world model (DriveDreamer-2) as an online restorer (DriveRestorer) and uses a progressive data update strategy to improve novel-trajectory rendering of Gaussian-splatting driving scene reconstruction for large lateral maneuvers.
Problem
Gaussian-splatting and NeRF driving scene reconstructions produce ghosting artifacts and fail when rendering large novel trajectories such as multi-lane shifts or lane changes, due to limited training data distribution.
Approach
Fine-tune a video world model to restore artifacted novel-trajectory renderings with masks emphasizing sky and distant regions, and progressively expand the novel trajectory during reconstruction training while mixing restored videos into the training set, rather than using a pretrained world model training-free as in DriveDreamer4D.
Key results
- ReconDreamer with Street Gaussians improves average NTA-IoU, NTL-IoU and FID over Street Gaussians by 24.87%, 6.72% and 29.97% relative (large-maneuver 6m shift focus).
- Against DriveDreamer4D with PVG, ReconDreamer achieves a 195.87% relative NTA-IoU gain on 6m lane shift and a 96.88% user-study win rate.
- On nuScenes 6m lane shift, NTA-IoU rises from 0.132 (Street Gaussians) to 0.246 for ReconDreamer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add online restoration and progressive data updates to Street Gaussians | Street Gaussians · Waymo novel trajectories, average | NTA-IoU: 0.456 -> 0.517 | Helped |
| add online restoration and progressive data updates to Street Gaussians | Street Gaussians · Waymo novel trajectories, average | NTL-IoU: 52.83 -> 54.60 | Helped |
| add online restoration and progressive data updates to Street Gaussians | Street Gaussians · Waymo novel trajectories, average | FID: 143.42 -> 105.55 | Helped |
| add ReconDreamer online restoration to PVG | PVG · Waymo Lane Shift @ 6m | NTA-IoU: 0.121 -> 0.358 | Helped |
| add ReconDreamer to Street Gaussians | Street Gaussians · nuScenes Lane Shift @ 6m | NTA-IoU: 0.132 -> 0.246 | Helped |
| add ReconDreamer to Street Gaussians | Street Gaussians · PandaSet No Shift | FID: 14.6 -> 16.1 | Hurt |
| add ReconDreamer to Street Gaussians | Street Gaussians · PandaSet Lane 2m | FID: 66.3 -> 65.4 | Hurt |
| add masks during DriveDreamer-2 fine-tuning (ablation) | DriveDreamer-2 · Waymo novel trajectories, average | NTA-IoU: 0.511 -> 0.517 | Helped |
| add masks during DriveDreamer-2 fine-tuning (ablation) | DriveDreamer-2 · Waymo novel trajectories, average | NTL-IoU: 54.42 -> 54.58 | Helped |
| add masks during DriveDreamer-2 fine-tuning (ablation) | DriveDreamer-2 · Waymo novel trajectories, average | FID: 114.95 -> 105.55 | Helped |
| set the progressive data update stride to 1.5m (ablation) | ReconDreamer with Street Gaussians · Waymo novel trajectories, average | NTA-IoU: 0.502 -> 0.517 | Helped |
| set the progressive data update stride to 1.5m (ablation) | ReconDreamer with Street Gaussians · Waymo novel trajectories, average | FID: 119.97 -> 105.55 | Helped |
Surprising findings
- Smaller stride values (0.5 and 1.0) in the progressive update reduce performance, and a stride of 6 disables the progressive update effect, yielding the worst NTA-IoU and NTL-IoU.
Limitations
- Evaluated on a small number of scenes (eight Waymo, eight nuScenes, ten PandaSet) with lateral shifts only.
- Relies on FID and IoU proxies plus a user study rather than direct ground truth for novel trajectories.
- Reconstruction still suffers ghosting in distant regions and sky when stride is poorly chosen.
Experimental setup
Datasets: Waymo nuScenes PandaSet
Benchmarks: Waymo nuScenes PandaSet
Metrics: NTA-IoU NTL-IoU FID user study win rate
Baselines: PVG S3Gaussian Deformable-GS Street Gaussians DriveDreamer4D UniSim NeuRAD Stable Diffusion Stable Video Diffusion
Builds on: DriveDreamer-2 DriveDreamer4D Street Gaussians Stable Video Diffusion PVG
Closest papers by meaning
- DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation (CVPR 2025 · similarity 0.89)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.87)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.86)
- DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving (NeurIPS 2025 · similarity 0.85)
- ReSim: Reliable World Simulation for Autonomous Driving (NeurIPS 2025 · similarity 0.85)
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.