Research map / 3D vision / 3D generation
UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds photorealistic, part-controllable 3D vehicle digital twins from a single urban image by retrieving Objaverse CAD models and optimizing Adobe procedural material graphs, then inserts them into 3DGS-reconstructed KITTI-360 scenes with fisheye-based lighting.
Problem
Autonomous driving simulation needs vehicle digital twins that are both photorealistic and controllable at part level, but reconstruction methods lack controllability and free CAD models lack high-quality materials, especially from single-view observations.
Approach
A retrieval-optimization pipeline: retrieve a part-disentangled CAD model via CLIP-aligned 3D embeddings, assign part-aware handcrafted procedural material graphs using semantic recognition (Grounded SAM on ControlNet-generated images), then optimize only albedo of metal and rubber graphs with differentiable rendering; scene insertion uses fisheye-derived spatially varying HDR lighting and 3DGS ba
Key results
- UrbanCAD achieves FID 62.80 and KID 0.0479 on 360-degree vehicle renderings, beating all baselines except Paint3D on KID (0.0417).
- On in-distribution synthetic scenes, Mask2Former (SwinL) reaches IOU 84.45 with UrbanCAD vs 73.15 for Wonder3D data.
- On door-opening out-of-distribution scenes, Mask2Former (SwinL) instance IOU drops from 95.67 on reference data to 85.44 on UrbanCAD OOD data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use UrbanCAD instead of Paint3D† | Vehicle 3D rendering · KITTI-360 360° renderings | FID: 67.52 -> 62.80 | Helped |
| use UrbanCAD instead of Paint3D† | Vehicle 3D rendering · KITTI-360 360° renderings | KID: 0.0417 -> 0.0479 | Hurt |
| use UrbanCAD instead of PhotoScene† | Vehicle 3D rendering · KITTI-360 matched-pose renderings | LPIPS: 0.5422 -> 0.5242 | Helped |
| add material optimization (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 1 | IOU: 69.21 -> 74.08 | Helped |
| add material optimization (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 2 | IOU: 68.69 -> 74.15 | Helped |
| add material optimization (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 3 | IOU: 67.06 -> 74.02 | Helped |
| add lighting estimation (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 1 | IOU: 73.11 -> 74.08 | Helped |
| add lighting estimation (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 2 | IOU: 73.33 -> 74.15 | Helped |
| add lighting estimation (ablation) | UrbanCAD · In-distribution driving scenarios, Trajectory 3 | IOU: 73.22 -> 74.02 | Helped |
Surprising findings
- Paint3D achieves better KID than UrbanCAD despite worse overall perceptual quality, attributed to rich but unrealistic textures.
- Retrieval alone (w/o material optimization) already outperforms reconstruction-based baselines on FID and KID.
Limitations
- Retrieved CAD geometry may differ from the vehicle in the input image due to semantic-based retrieval.
- Spatially varying lighting may be inaccurate when insertion position is far from fisheye cameras.
- Only albedo of metal and rubber is optimized; glass is directly assigned without optimization.
Experimental setup
Datasets: KITTI-360 Objaverse (26k+ car and vehicle models) Adobe 3D Asset Library (13k+ materials) MVMC (Multi-View Marketplace Cars) Real-world car datasets for FID/KID
Benchmarks: KITTI-360 MVMC
Metrics: FID KID LPIPS IOU PQ instance-level IOU
Baselines: PixelNeRF Wonder3D LRM HUGS LatentPaint Paint3D PhotoScene
Builds on: MATch DiffMat v2 3D Gaussian Splatting InvRenderNet PhotoScene
Closest papers by meaning
- ProtoCar: Learning 3D Vehicle Prototypes from Single-View and Unconstrained Driving Scene Images (AAAI 2025 · similarity 0.88)
- SceneCrafter: Controllable Multi-View Driving Scene Editing (CVPR 2025 · similarity 0.87)
- X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability (NeurIPS 2025 · similarity 0.86)
- DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation (CVPR 2025 · similarity 0.85)
- StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models (CVPR 2025 · similarity 0.85)
- SceneDiffuser++: City-Scale Traffic Simulation via a Generative World Model (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.