Research map / 3D vision / 3D generation
WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WildFusion, a 3D-aware latent diffusion model trained on unposed in-the-wild images by modeling instances in view space, using a depth-supervised autoencoder with triplanes and adversarial novel-view supervision.
Problem
3D-aware image synthesis typically requires posed images or a shared canonical camera system, which is hard or impossible to define for in-the-wild datasets; existing GAN-based view-space methods produce flat geometry and poor distribution coverage.
Approach
Model instances in view space with a fixed input camera, train a 3D-aware autoencoder (FPN encoder, transformer+CNN decoder to triplanes) with reconstruction, adversarial novel-view and monocular-depth losses, then train a latent diffusion model on its 2D latent grid to get a non-GAN 3D-aware generator.
Key results
- On SDIP Dogs, WildFusion reaches FID 12.2 and NFS 31.7 vs. 3DGP's FID 5.9 and NFS 36.3, and beats EG3D* and StyleNeRF on FID and NFS.
- On SDIP Elephants, WildFusion achieves FID 2.9 and recall 0.34, outperforming the baselines on FID and recall.
- On ImageNet (s=2), WildFusion achieves NFS 33.8, precision 0.59 and recall 0.20, well above GAN baselines, though FID (35.4) trails 3DGP (19.7); 3DGP shows strong mode collapse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use WildFusion instead of EG3D* on SDIP Dogs | 3D-aware image synthesis · SDIP Dogs | FID_CLIP: 5.8 -> 5.2 | Helped |
| use WildFusion instead of EG3D* on SDIP Elephants | 3D-aware image synthesis · SDIP Elephants | Recall: 0.31 -> 0.34 | Helped |
| use WildFusion instead of StyleNeRF on SDIP Horses | 3D-aware image synthesis · SDIP Horses | FID: 4.5 -> 4.3 | Helped |
| use WildFusion instead of 3DGP on ImageNet | 3D-aware image synthesis · ImageNet | Recall: 0.02 -> 0.20 | Helped |
| use WildFusion instead of 3DGP on ImageNet | 3D-aware image synthesis · ImageNet | FID: 19.7 -> 35.4 | Hurt |
| add novel-view discriminator Dφ (ablation) | WildFusion autoencoder · SDIP Dogs ablation | nvFID: 53.2 -> 61.1 | Hurt |
| replace the fully convolutional decoder with a ViT/convolution combination (ablation) | WildFusion autoencoder · SDIP Dogs ablation | nvFID: 61.1 -> 48.3 | Helped |
| add depth discriminator Ddepthχ (ablation) | WildFusion autoencoder · SDIP Dogs ablation | nvFID: 48.3 -> 40.5 | Helped |
| model unbounded scenes with disparity sampling and coordinate contraction (ablation) | WildFusion autoencoder · SDIP Dogs ablation | nvFID: 40.5 -> 32.8 | Helped |
| add 2D rendered-depth loss (ablation) | WildFusion autoencoder · SDIP Dogs ablation | nvFID: 32.8 -> 34.6 | No clear effect |
| provide monocular depth as encoder input (ablation) | WildFusion autoencoder · SDIP Dogs ablation | NFS: 32.0 -> 33.3 | Helped |
| remove 3D rendering-weight depth supervision Ldepth³D (ablation) | WildFusion autoencoder · SDIP Dogs ablation | NFS: 33.7 -> 30.8 | Hurt |
Surprising findings
- 3DGP achieves better FID on ImageNet (19.7) than WildFusion, yet shows near-zero recall and mode collapse, illustrating FID's limitations here.
- Adding a monocular depth discriminator to EG3D* did not consistently improve results (e.g., worse FID on Dogs and Elephants).
Limitations
- ImageNet model trained with total batch size 256 due to compute limits, likely underperforming relative to typical diffusion models.
- Relies on monocular depth estimates defined only up to scale, and on fixed camera intrinsics and a single fixed input view.
- Evaluated only at 256x256 resolution and on unconditional/class-conditional settings; text-conditioned scaling left to future work.
Experimental setup
Datasets: SDIP Dogs SDIP Elephants SDIP Horses ImageNet
Benchmarks: SDIP Dogs SDIP Elephants SDIP Horses ImageNet
Metrics: LPIPS nvFID FID FID_CLIP NFS Precision Recall
Baselines: EG3D* EG3D* + depth discriminator 3DGP POF3D StyleNeRF IVID SceneScape
Builds on: latent diffusion models EG3D 3DGP StyleGAN3 feature pyramid network
Closest papers by meaning
- Efficient-3Dim: Learning a Generalizable Single-image Novel-view Synthesizer in One Day (ICLR 2024 · similarity 0.89)
- Denoising Diffusion via Image-Based Rendering (ICLR 2024 · similarity 0.89)
- A 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale (NeurIPS 2026 · similarity 0.89)
- LT3SD: Latent Trees for 3D Scene Diffusion (CVPR 2025 · similarity 0.89)
- CAD: Photorealistic 3D Generation via Adversarial Distillation (CVPR 2024 · similarity 0.89)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.