Research map / 3D vision / 3D generation
GenAssets: Generating in-the-wild 3D Assets in Latent Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenAssets, a "reconstruct-then-generate" pipeline that learns a latent space of occlusion-aware neural actor assets from in-the-wild PandaSet driving data and trains a latent diffusion model over it to generate complete 360° vehicle/pedestrian assets.
Problem
Generating complete, high-quality 3D assets of traffic actors for multi-sensor simulation from sparse, occluded, in-the-wild driving data, where existing per-scene reconstruction is slow and generalizes poorly to novel views, and generative models trained on clean synthetic objects degrade on real d
Approach
Jointly learn a KL-regularized latent code per actor (plus class embedding) decoded to triplane neural fields via occlusion-aware compositional volume rendering over many scenes, then train a latent diffusion model on these codes so generation and single-view/LiDAR-guided reconstruction operate in a compact asset latent space.
Key results
- Sparse view synthesis on PandaSet (10% frames for training): PSNR 21.34, SSIM 0.825, LPIPS 0.113, outperforming NeuRAD (21.07 / 0.825 / 0.129).
- Unconditional generation FID 59.50 and KID 28.32 (x10^3) vs. EG3D FID 80.56 and DiscoScene 138.48.
- Data augmentation with generated assets raises BEVFormer mAP from 27.08 to 29.32 on PandaSet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenAssets instead of NeuRAD for sparse view synthesis | PandaSet reconstruction · PandaSet sparse view synthesis | PSNR: 21.07 -> 21.34 | Helped |
| Use GenAssets instead of NeuRAD for sparse view synthesis | PandaSet reconstruction · PandaSet sparse view synthesis | SSIM: 0.825 -> 0.825 | No clear effect |
| Use GenAssets instead of NeuRAD for sparse view synthesis | PandaSet reconstruction · PandaSet sparse view synthesis | LPIPS: 0.129 -> 0.113 | Helped |
| Use GenAssets instead of Street Gaussian for novel camera synthesis | PandaSet reconstruction · PandaSet novel camera synthesis | PSNR: 17.81 -> 18.36 | Helped |
| Use GenAssets instead of G3R for novel camera synthesis | PandaSet reconstruction · PandaSet novel camera synthesis | SSIM: 0.723 -> 0.805 | Helped |
| Use GenAssets instead of NeuRAD for novel camera synthesis | PandaSet reconstruction · PandaSet novel camera synthesis | LPIPS: 0.214 -> 0.147 | Helped |
| Use GenAssets instead of NeuRAD for 360° view synthesis | PandaSet reconstruction · PandaSet 360° view synthesis | FID: 159.38 -> 100.28 | Helped |
| Use GenAssets instead of EG3D for unconditional generation | PandaSet 3D asset generation · PandaSet unconditional generation | FID: 80.56 -> 59.50 | Helped |
| Use GenAssets instead of EG3D for unconditional generation | PandaSet 3D asset generation · PandaSet unconditional generation | KID: 38.97 -> 28.32 | Helped |
| Add GenAssets-generated actors to the training data | BEVFormer · PandaSet 3D object detection | mAP: 27.08 -> 29.32 | Helped |
| Add GenAssets-generated actors to the training data | BEVFormer · PandaSet 3D object detection | AP@1m: 8.58 -> 9.78 | Helped |
| Add GenAssets-generated actors to the training data | BEVFormer · PandaSet 3D object detection | AP@2m: 26.99 -> 29.18 | Helped |
Surprising findings
- Per-scene NeRF/Gaussian reconstruction methods and generalizable reconstructors perform worse than a generative approach even in the sparse-view setting.
Limitations
- Evaluated only on PandaSet, a single city dataset with 103 scenes.
- Generated assets lack dynamic lighting and animation, as noted in future work.
- Object-level evaluation relies on instance masks produced by a visual foundation model.
Experimental setup
Datasets: PandaSet
Benchmarks: PandaSet
Metrics: PSNR SSIM LPIPS FID KID mAP AP@1m AP@2m AP@4m
Baselines: NeuRAD Street Gaussians G3R PixelSplat EG3D DiscoScene SSDNeRF MeshLRM
Builds on: NeuRAD EG3D SSDNeRF latent diffusion models DDIM
Closest papers by meaning
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.87)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.87)
- Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata (CVPR 2024 · similarity 0.87)
- Structured 3D Latents for Scalable and Versatile 3D Generation (CVPR 2025 · similarity 0.87)
- Genesis: Multimodal Driving Scene Generation with Spatio-Temporal and Cross-Modal Consistency (NeurIPS 2025 · similarity 0.87)
- Glad: A Streaming Scene Generator for Autonomous Driving (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.