Research map / 3D vision / NeRF and radiance fields
Latent Radiance Fields with 3D-aware 2D Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a latent radiance field (3DGS in VAE latent space) by fine-tuning the Stable Diffusion VAE with a correspondence-consistency loss and then aligning the decoder to LRF renderings, for photorealistic NVS from 2D latents.
Problem
Radiance-field reconstruction in the 2D latent space of VAEs suffers from a domain gap: latents lack 3D consistency and high-frequency noise, so rendered latents decode to degraded images.
Approach
Inject 3D awareness into the VAE latent space via a multi-view correspondence consistency loss (pose-weighted L1 between latents of matched pixels) plus a regularizer to the original VAE, then fit 3DGS directly on latents (LRF) and fine-tune the decoder on LRF renderings (VAE-RF alignment) to close the decoding gap.
Key results
- On DL3DV-10K, 3DGS-LRF reaches 22.45 PSNR / 0.667 SSIM / 0.197 LPIPS, versus 21.81 PSNR for Mip-Splatting on images and 20.57 PSNR for the fine-tuned-free 3DGS-VAE latent baseline.
- On MVImgNet (out-of-distribution), LRF achieves 26.26 PSNR and 0.178 LPIPS, the best among compared methods.
- Ablation on DL3DV-10K shows encoder-only fine-tuning gives 21.16 PSNR, decoder-only 21.73, and both together 22.45 over the 20.57 VAE baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · MVImgNet | PSNR: 24.89 -> 26.26 | Helped |
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · NeRF-LLFF | SSIM: 0.493 -> 0.541 | Helped |
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · DL3DV-10K | LPIPS: 0.346 -> 0.197 | Helped |
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · Mip-NeRF360 | PSNR: 22.38 -> 20.83 | Hurt |
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · Mip-NeRF360 | SSIM: 0.502 -> 0.469 | Hurt |
| use 3DGS-LRF instead of the strongest baseline | 3DGS-LRF · Mip-NeRF360 | LPIPS: 0.432 -> 0.328 | Helped |
| add correspondence-aware encoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | PSNR: 20.57 -> 21.16 | Helped |
| add correspondence-aware encoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | LPIPS: 0.346 -> 0.282 | Helped |
| add VAE-RF aligned decoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | PSNR: 20.57 -> 21.73 | Helped |
| add VAE-RF aligned decoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | LPIPS: 0.346 -> 0.208 | Helped |
| add correspondence-aware encoder fine-tuning and VAE-RF aligned decoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | PSNR: 20.57 -> 22.45 | Helped |
| add correspondence-aware encoder fine-tuning and VAE-RF aligned decoder fine-tuning (ablation) | 3DGS-LRF · DL3DV-10K | SSIM: 0.595 -> 0.667 | Helped |
Surprising findings
- Fine-tuning the VAE encoder/decoder reduces VAE reconstruction PSNR on DL3DV-10K (24.59 to 23.25), a small cost to image reconstruction.
- Latent-NeRF adapted to NVS performs worse than the plain 3DGS-VAE latent baseline on most datasets.
Limitations
- Latent-space NVS trades off some VAE reconstruction fidelity (PSNR drops on DL3DV-10K after fine-tuning).
- Requires precomputed COLMAP correspondences and per-scene LRF training to build the decoder-alignment dataset.
- Evaluated only with 3DGS as the radiance field; other compact representations are left to future work.
Experimental setup
Datasets: DL3DV-10K MVImgNet NeRF-LLFF Mip-NeRF360
Benchmarks: DL3DV-10K MVImgNet NeRF-LLFF Mip-NeRF360
Metrics: PSNR SSIM LPIPS
Baselines: 3DGS Mip-Splatting 3DGS-VAE Latent-NeRF Feature-GS GSGEN DreamFusion
Builds on: 3D Gaussian Splatting Latent Diffusion Models VAE Latent-NeRF Feature 3DGS
Closest papers by meaning
- Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space (AAAI 2025 · similarity 0.88)
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (CVPR 2024 · similarity 0.87)
- OpenHype: Hyperbolic Embeddings for Hierarchical Open-Vocabulary Radiance Fields (NeurIPS 2025 · similarity 0.87)
- NeRFPrior: Learning Neural Radiance Field as a Prior for Indoor Scene Reconstruction (CVPR 2025 · similarity 0.87)
- HyRF: Hybrid Radiance Fields for Memory-efficient and High-quality Novel View Synthesis (NeurIPS 2025 · similarity 0.86)
- Mask-Based Modeling for Neural Radiance Fields (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.