Research map / 3D vision / Gaussian splatting
3D StreetUnveiler with Semantic-aware 2DGS - a simple baseline
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs an empty street from in-car video by building a hard-label semantic 2DGS, generating inpainting masks from rendered alpha maps, and inpainting frames in reverse time order with LeftRefill to supervise 2DGS re-optimization.
Problem
Removing all temporarily static objects (parked cars, standing pedestrians) from long-trajectory in-car camera videos to reconstruct a 3D empty street, where inpainting masks for completely unobservable regions and temporal consistency across long trajectories are hard to obtain.
Approach
Represent the street with hard-label semantic 2D Gaussians, use semantic distortion and shrinking losses to make fully unobservable regions have low alpha (automatic inpainting masks), and inpaint frames in reverse time so later frames guide earlier ones via reference-based inpainting (high-to-low resolution), then re-optimize 2DGS on the pseudo-labels.
Key results
- On Waymo, the method reaches LPIPS 0.216 and FID 127.581, beating SPIn-NeRF (in 2DGS) at 0.221/140.831 and Infusion at 0.307/176.882 on LPIPS and FID; only SDXL has lower FID (116.634).
- On Pandaset, LPIPS 0.261 and FID 155.527 versus SPIn-NeRF 0.266/174.223 and ProPainter 0.286/178.135.
- Time-reversal inpainting improves over time-forward inpainting (Waymo LPIPS 0.216 vs 0.220; FID 127.581 vs 136.858).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use StreetUnveiler rather than SPIn-NeRF (2DGS) | Street inpainting · Waymo | LPIPS: 0.221 -> 0.216 | Helped |
| use StreetUnveiler rather than SDXL | Street inpainting · Waymo | FID: 116.634 -> 127.581 | Hurt |
| use StreetUnveiler rather than SPIn-NeRF (2DGS) | Street inpainting · Pandaset | LPIPS: 0.266 -> 0.261 | Helped |
| use StreetUnveiler rather than SDXL | Street inpainting · Pandaset | FID: 133.042 -> 155.527 | Hurt |
| use LaMa pseudo-labels instead of the proposed pseudo-label pipeline (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.226 | Hurt |
| use SDXL pseudo-labels instead of the proposed pseudo-label pipeline (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.222 | Hurt |
| use ProPainter pseudo-labels instead of the proposed pseudo-label pipeline (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.224 | Hurt |
| replace 2DGS with 3DGS (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.219 | Hurt |
| replace 2DGS with 3DGS (ablation) | StreetUnveiler · Pandaset | LPIPS: 0.261 -> 0.280 | Hurt |
| use time-forward rather than time-reversal inpainting (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.220 | Hurt |
| use time-forward rather than time-reversal inpainting (ablation) | StreetUnveiler · Pandaset | LPIPS: 0.261 -> 0.270 | Hurt |
| use naive reverse-order LeftRefill rather than the full pipeline (ablation) | StreetUnveiler · Waymo | LPIPS: 0.216 -> 0.227 | Hurt |
Surprising findings
- Single-image SDXL achieves the lowest FID on both datasets, outperforming the proposed method on that metric despite inconsistent video frames.
- Infusion's Pandaset FID is reported identically to its Waymo FID (176.882), which may indicate a reporting error.
Limitations
- Reconstruction stage dominates runtime (~10k seconds per scene).
- Time-reversal framework assumes forward-facing cameras; side and back views lack consistency guarantees.
- Only static objects are removed; dynamic objects need separate handling; geometry Chamfer Distance is worse than StreetSurf.
Experimental setup
Datasets: Waymo Open Perception Dataset Pandaset
Benchmarks: Waymo Open Perception Dataset Pandaset
Metrics: LPIPS FID Chamfer Distance F-Score
Baselines: LaMa SDXL ProPainter SPIn-NeRF (in 2DGS) Infusion LeftRefill
Builds on: 2D Gaussian Splatting LeftRefill SPIn-NeRF 3D Gaussian Splatting
Closest papers by meaning
- HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting (CVPR 2024 · similarity 0.88)
- DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes (CVPR 2025 · similarity 0.88)
- Effective Multi-sensor Conditioning for Street-view Novel-view Synthesis (NeurIPS 2026 · similarity 0.87)
- D$^2$GS: Dense Depth Regularization for LiDAR-free Urban Scene Reconstruction (NeurIPS 2025 · similarity 0.86)
- EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis (CVPR 2025 · similarity 0.86)
- EGSRAL:An Enhanced 3D Gaussian Splatting Based Renderer with Automated Labeling for Large-Scale Driving Scene (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.