Research map / 3D vision / Gaussian splatting
VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VR-Drive, an end-to-end driving framework that jointly learns feed-forward 3D Gaussian Splatting for planning-aware novel-view augmentation, plus a viewpoint-mixed memory bank and viewpoint-consistent distillation, and introduces a nuScenes-based benchmark with unseen camera viewpoints.
Problem
End-to-end autonomous driving planners degrade when camera viewpoints (pitch, height, depth) differ from those seen in training, which is common across vehicle rigs.
Approach
Jointly train a feed-forward 3DGS reconstruction head inside the E2E-AD pipeline to synthesize random novel viewpoints online as training augmentation, then stabilize them with a viewpoint-mixed memory bank and distillation of original-view instance features into novel-view features, without extra annotations.
Key results
- On nuScenes unseen-viewpoint average, VR-Drive reaches 0.68 m average L2 and 0.11% collision rate versus DiffusionDrive's 1.17 m and 0.41%.
- On CARLA Town05-Nov unseen average, VR-Drive achieves 88.25 Driving Score and 98.28 Route Completion versus baseline 48.25 DS and 94.87 RC.
- Ablation shows adding the novel-view input (ID-3) gives the largest gain on unseen average collision rate reduction from 0.30% to 0.19%, and the full model reaches 0.11%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly learn scene reconstruction (ablation) | VR-Drive · nuScenes Seen | L2 Avg. (m): 0.63 -> 0.59 | Helped |
| use generated novel views as perception and planning input (ablation) | VR-Drive with scene reconstruction · nuScenes Unseen Average | L2 Avg. (m): 0.90 -> 0.79 | Helped |
| add viewpoint-mixed memory bank (ablation) | VR-Drive · nuScenes Unseen Average | L2 Avg. (m): 0.73 -> 0.68 | Helped |
| add cyclic reconstruction loss (ablation) | VR-Drive · nuScenes Unseen Average | Collision Avg. (%): 0.16 -> 0.11 | Helped |
| add viewpoint-consistent distillation (ablation) | VR-Drive · nuScenes Unseen Average | Collision Avg. (%): 0.14 -> 0.11 | Helped |
| expand training-time viewpoint range to Superset (ablation) | VR-Drive · nuScenes Unseen Average | L2 Avg. (m): 0.68 -> 0.68 | No clear effect |
| narrow training-time viewpoint range to Subset (ablation) | VR-Drive · nuScenes Unseen Average | L2 Avg. (m): 0.68 -> 0.82 | Hurt |
| use VR-Drive instead of DiffusionDrive under unseen camera viewpoints | nuScenes E2E-AD · nuScenes Unseen Average | L2 Avg. (m): 1.17 -> 0.68 | Helped |
| use VR-Drive instead of the baseline | CARLA Town05-Nov · Town05-Nov Unseen Average | Driving Score (DS): 48.25 -> 88.25 | Helped |
Surprising findings
- Joint scene reconstruction alone (without novel view input) improves seen-view planning L2 and collision rate, suggesting 3D geometric learning helps even the original viewpoint.
- Training with novel-view augmentation did not trade off original-view performance; the full model matches or improves on the original view.
Limitations
- Performance depends on accuracy of camera calibration; calibration errors can degrade results.
- Unseen-viewpoint nuScenes benchmark relies on offline 3DGS renderings, which required excluding 4 of 150 sequences for quality issues.
Experimental setup
Datasets: nuScenes CARLA Town01-04, 06, 07, 10 training routes
Benchmarks: nuScenes open-loop planning with unseen viewpoints CARLA Town05-Nov closed-loop
Metrics: L2 (m) collision rate (%) Driving Score Route Completion
Baselines: AD-MLP BEV-Planner VAD SparseDrive DiffusionDrive ST-P3 TCP
Builds on: 3D Gaussian Splatting SparseDrive DiffusionDrive ST-P3
Closest papers by meaning
- VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving (CVPR 2026 · similarity 0.90)
- Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution (NeurIPS 2025 · similarity 0.88)
- FreeVS: Generative View Synthesis on Free Driving Trajectory (ICLR 2025 · similarity 0.88)
- DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving (ICLR 2025 · similarity 0.88)
- UniDrive: Towards Universal Driving Perception Across Camera Configurations (ICLR 2025 · similarity 0.88)
- Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.