Research map / Image generation / Autoregressive and latent generation
Bringing NeRFs to the Latent Space: Inverse Graphics Autoencoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IG-AE, an image autoencoder whose latent space is regularized with jointly trained tri-plane latent 3D scenes so NeRFs can be trained in latent space, and releases a Nerfstudio extension for latent NeRF training.
Problem
Image autoencoder latent spaces lack 3D consistency across views, so NeRFs cannot be trained directly on latent images, limiting speed gains and interoperability with latent-based 2D methods.
Approach
Regularize an image autoencoder by aligning its encoded latents with renderings of jointly trained latent tri-plane scenes (3D regularization), plus autoencoder-preservation losses on synthetic and real images, yielding a 3D-aware latent space in which NeRFs can be trained via latent supervision followed by RGB alignment.
Key results
- On ShapeNet Bags, Vanilla-NeRF with IG-AE reaches 29.57 PSNR vs 28.81 with standard AE; TensoRF improves from 26.19 to 28.40 PSNR.
- Latent Vanilla-NeRF training takes 28 min vs 637 min for RGB training on a single L4 GPU, with rendering 19.7 ms vs 1201 ms, but NVS PSNR drops to 33.60 from 40.78.
- IG-AE improves Objaverse-scene latent NVS for Vanilla-NeRF to 50.31 PSNR vs 38.36 for the AE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IG-AE instead of a standard AE with Vanilla-NeRF | Latent NeRF · ShapeNet Bags, Hats, and Vases | PSNR: 28.81 -> 29.57; 28.75 -> 29.31; 32.00 -> 33.06 | Helped |
| Use IG-AE instead of a standard AE with Vanilla-NeRF | Latent NeRF · ShapeNet Bags, Hats, and Vases | LPIPS: 0.046 -> 0.044; 0.032 -> 0.033; 0.023 -> 0.021 | Mixed |
| Remove 3D regularization (ablation) | IG-AE with Vanilla-NeRF · Objaverse Cake and Figurine NVS | PSNR: 34.38 -> 32.09; 33.17 -> 30.94 | Hurt |
| Add 3D regularization (ablation) | IG-AE with Vanilla-NeRF · Objaverse Cake NVS | PSNR: 32.09 -> 34.38 | Helped |
| Remove AE preservation (ablation) | IG-AE · ImageNet held-out reconstruction | PSNR: 29.57 -> 17.66 | Hurt |
| Add AE preservation (ablation) | IG-AE · ImageNet held-out reconstruction | PSNR: 17.66 -> 29.57 | Helped |
| Remove 3D regularization (ablation) | IG-AE · ImageNet held-out reconstruction | PSNR: 29.83 -> 29.57 | No clear effect |
| Add RGB alignment after latent supervision (ablation) | IG-AE latent Vanilla-NeRF · Objaverse Cake | PSNR: 24.34 -> 34.68 | Helped |
| Use kl-f4 instead of kl-f16 (ablation) | Latent NeRF · Latent NVS across evaluated scenes | PSNR: 29.58 -> 35.38; 23.54 -> 30.03; 25.91 -> 33.56; 27.59 -> 33.42 | Helped |
| Train in IG-AE latent space instead of RGB space | Vanilla-NeRF · Objaverse Cake, House, and Figurine | PSNR: 40.78 -> 33.60 | Hurt |
Surprising findings
- Latent NeRFs trained on IG-AE have a clear high-frequency detail loss, with LPIPS slightly worse than AE for autoencoding despite better PSNR/SSIM.
Limitations
- Latent NeRFs lose high-frequency details, so NVS is lower than RGB NeRFs.
- Evaluation limited to simple object-level synthetic scenes; not applied to complex real-world scenes.
- Limited to 128x128 object-level experiments with a single autoencoder choice.
Experimental setup
Datasets: Objaverse ImageNet ShapeNet Hats ShapeNet Bags ShapeNet Vases
Benchmarks: ShapeNet Hats ShapeNet Bags ShapeNet Vases Objaverse held-out scenes (Cake, House, Figurine, Helmet, Book, Burger) ImageNet reconstruction
Metrics: PSNR SSIM LPIPS training time rendering time
Baselines: AE (standard autoencoder latent) IG-AE (no Pr) IG-AE (no 3D) RGB-space NeRFs
Builds on: Nerfstudio Vanilla NeRF Instant-NGP TensoRF K-Planes
Closest papers by meaning
- Improving the Diffusability of Autoencoders (ICML 2025 · similarity 0.87)
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment (NeurIPS 2024 · similarity 0.86)
- Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective (NeurIPS 2024 · similarity 0.86)
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields (CVPR 2025 · similarity 0.86)
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.85)
- Masked Autoencoders Are Effective Tokenizers for Diffusion Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.