Research map / 3D vision / NeRF and radiance fields
In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-N-Out, a 3D GAN inversion method that models out-of-distribution face content (makeup, occluders) with a second tri-plane radiance field composited with the EG3D in-distribution field, enabling faithful reconstruction and InD-only face editing.
Problem
GAN inversion with 3D-aware GANs (EG3D pre-trained on FFHQ) struggles to reconstruct out-of-distribution objects such as heavy makeup or occluders, creating a trade-off between reconstruction fidelity and editability.
Approach
Decompose the image into two radiance fields: an in-distribution EG3D-inverted face (W+ latent) and a separate OOD tri-plane radiance field with per-frame latent codes, combined via composite volume rendering with a learned blending weight and entropy/latent regularizers; edits are applied only to the in-distribution latent.
Key results
- Reconstruction on images: LPIPS 0.1106, PSNR 19.86, SSIM 0.8175, ID similarity 0.9685, beating HFGI3D, GOAE, PTI, and W+ on all metrics in Table 1.
- Editing identity preservation averaged over edits: 0.9511 on images and 0.9177 on videos, the best among compared methods (Table 2).
- Ablation shows removing blending-weight loss Lb or latent regularizer Lw lowers editability (ID similarity 0.9070 and 0.9024 vs 0.9177 full).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use In-N-Out's composed InD/OOD radiance-field inversion instead of E3DGE | EG3D inversion · Images | LPIPS: 0.1709 -> 0.1106 | Helped |
| Use In-N-Out's composed InD/OOD radiance-field inversion instead of E3DGE | EG3D inversion · Images | PSNR: 15.28 -> 19.86 | Helped |
| Use In-N-Out's composed InD/OOD radiance-field inversion instead of PTI | EG3D video inversion · Videos | LPIPS: 0.3144 -> 0.2237 | Helped |
| Use In-N-Out's composed InD/OOD radiance-field inversion instead of GOAE | EG3D video inversion · Videos | SSIM: 0.6470 -> 0.7052 | Helped |
| Add blending-weight regularization Lb (ablation) | In-N-Out · Videos | L2: 0.0322 -> 0.0339 | Hurt |
| Add blending-weight regularization Lb (ablation) | In-N-Out · Videos | ID similarity: 0.9070 -> 0.9177 | Helped |
| Add latent-code regularizer Lw (ablation) | In-N-Out · Videos | LPIPS: 0.2238 -> 0.2237 | No clear effect |
| Add latent-code regularizer Lw (ablation) | In-N-Out · Videos | ID similarity: 0.9024 -> 0.9177 | Helped |
Surprising findings
- Reconstruction with a second OOD radiance field trades off computation: the method takes about 2.68h on images versus 1.45h for PTI and 0.47h for W optimization, though it is more accurate.
Limitations
- Editing within OOD-dominated regions is hard (e.g., adding eyeglasses where the OOD blending weight is near 1).
- Fails under extreme poses such as side views and can produce duplicate objects when the OOD item is glasses.
- Video results may show temporal inconsistency and floaters with limited object movement.
Experimental setup
Datasets: 20 in-the-wild online face videos with OOD content (Creative Commons)
Benchmarks: in-house 20-video OOD face dataset
Metrics: LPIPS PSNR SSIM ID similarity L2 runtime
Baselines: HFGI3D PTI W+ optimization W optimization VIVE3D GOAE IDE-3D E3DGE
Builds on: EG3D PTI ChunkyGAN InterfaceGAN StyleCLIP
Closest papers by meaning
- 3D GAN Inversion With Facial Symmetry Prior (CVPR 2023 · similarity 0.91)
- Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion (CVPR 2023 · similarity 0.91)
- High-Fidelity GAN Inversion for Image Attribute Editing (CVPR 2022 · similarity 0.91)
- High-Fidelity 3D GAN Inversion by Pseudo-Multi-View Optimization (CVPR 2023 · similarity 0.90)
- Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images (NeurIPS 2024 · similarity 0.89)
- WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.