Research map / Image generation / Editing and style transfer
GenN2N: Generative NeRF2NeRF Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenN2N, a unified NeRF-to-NeRF translation framework that lifts edits from any plug-and-play 2D image-to-image translator into NeRF via a conditional 3D VAE-GAN with contrastive regularization over sampled edit codes.
Problem
Existing NeRF editing methods are task-specific and, when lifting inconsistent multi-view 2D edits into 3D, produce blurry or deterministic results that cannot capture the diversity of possible edits.
Approach
Treat multi-view 2D edits as samples from a distribution of 3D edits: encode each edited image into a 64-d edit code (frozen Stable Diffusion VAE encoder plus tiny MLP), condition a translated NeRF on that code, and train it with reconstruction, conditional adversarial, KL, and contrastive losses so that sampling codes yields diverse, multi-view consistent 3D edits.
Key results
- Text-driven editing: GenN2N reaches CLIP direction consistency 0.9864 and FID 137.740, versus 0.9828 and 148.021 for Instruct-NeRF2NeRF.
- Super-resolution on LLFF: PSNR 28.501 and SSIM 0.913 vs 27.957 and 0.897 for NeRF-SR.
- Inpainting on the statue dataset: PSNR 26.868 vs 24.369 for SPIn-NeRF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GenN2N instead of Instruct-NeRF2NeRF | NeRF text-driven editing · Text-driven editing | FID: 148.021 -> 137.740 | Helped |
| use GenN2N instead of PaletteNeRF | NeRF colorization · Colorization | CF: 39.654 -> 65.099 | Helped |
| use GenN2N instead of NeRF-SR | NeRF super-resolution · Super-resolution | PSNR: 27.957 -> 28.501 | Helped |
| use GenN2N instead of SPIn-NeRF | NeRF inpainting · Inpainting | PSNR: 24.369 -> 26.868 | Helped |
| add adversarial loss (ablation) | GenN2N · Text-driven editing | FID: 162.275 -> 137.740 | Helped |
| add contrastive loss (ablation) | GenN2N · Text-driven editing | FID: 156.524 -> 137.740 | Helped |
| add adversarial loss (ablation) | GenN2N · Colorization | FID: 137.740 -> 35.041 | Helped |
| add contrastive loss (ablation) | GenN2N · Colorization | FID: 105.750 -> 35.041 | Helped |
| add adversarial loss (ablation) | GenN2N · Super-resolution | LPIPS: 0.2001 -> 0.0748 | Helped |
| add contrastive loss (ablation) | GenN2N · Super-resolution | LPIPS: 0.1834 -> 0.0748 | Helped |
Surprising findings
- Instruct-NeRF2NeRF trained twice yields nearly identical results, while GenN2N produces diverse outputs by sampling edit codes.
Limitations
- Relies on 2D translators that produce multi-view inconsistent edits, requiring the generative model to absorb inconsistency.
- Ablation shows blurriness when contrastive loss is removed, attributed to pose information leaking into the edit code.
Experimental setup
Datasets: Face (Instruct-NeRF2NeRF) Fangzhou self-portrait Farm (Instruct-NeRF2NeRF) LLFF statue dataset (SPIn-NeRF)
Benchmarks: Face Fangzhou Farm LLFF statue dataset
Metrics: CLIP text-image direction similarity CLIP direction consistency FID colorfulness score PSNR SSIM LPIPS
Baselines: Instruct-NeRF2NeRF InstructPix2Pix+NeRF PaletteNeRF NeRF-SR SPIn-NeRF DDColor+NeRF ResShift+NeRF LaMa+NeRF
Builds on: InstructPix2Pix Instruct-NeRF2NeRF Nerfstudio Stable Diffusion VAE Blended Latent Diffusion
Closest papers by meaning
- ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF (ICLR 2024 · similarity 0.89)
- E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.86)
- LLaNA: Large Language and NeRF Assistant (NeurIPS 2024 · similarity 0.86)
- Perturb-and-Revise: Flexible 3D Editing with Generative Trajectories (CVPR 2025 · similarity 0.85)
- HyperFields: Towards Zero-Shot Generation of NeRFs from Text (ICML 2024 · similarity 0.85)
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.