Research map / 3D vision / 3D generation
Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a dual-encoder 3D GAN inversion framework on PanoHead that stitches a reconstruction-focused encoder and an adversarially trained encoder in the triplane domain, using an occlusion-aware triplane discriminator.
Problem
Single-image 3D GAN inversion into PanoHead's latent space must balance high-fidelity reconstruction of the visible input view against realistic prediction of occluded head regions for full 360-degree rendering; prior encoders are built on EG3D and target near-frontal views.
Approach
Train two encoders (one for same-view fidelity, one with an adversarial loss from an occlusion-aware triplane discriminator trained only on occluded triplane features) and combine them by taking visible regions from Encoder 1 and occluded regions from Encoder 2 via occlusion masks.
Key results
- On FFHQ+LPFF, the method reaches LPIPS 0.10, ID 0.87 and FID 65.44, matching the best encoder on reconstruction and far better FID than GOAE (168.96) and TriplaneNetv2 (98.97).
- Inference takes 0.37 s per sample versus 86.52 s for PTI, about 250x faster, with comparable reconstruction quality to PTI (LPIPS 0.11, ID 0.89).
- On multi-view MEAD (±60°), LPIPS 0.223 and ID 0.706 beat all encoder and optimization baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the dual-encoder method rather than GOAE | PanoHead inversion · FFHQ + LPFF | FID: 168.96 -> 65.44 | Helped |
| Use the dual-encoder method rather than TriplaneNetv2 | PanoHead inversion · FFHQ + LPFF | FID: 98.97 -> 65.44 | Helped |
| Use the dual-encoder method rather than PTI | PanoHead inversion · FFHQ + LPFF | FID: 59.40 -> 65.44 | Hurt |
| Use the dual-encoder method rather than pSp | PanoHead inversion · MEAD ±60° | LPIPS: 0.245 -> 0.223 | Helped |
| Use the dual-encoder method rather than pSp | PanoHead inversion · MEAD ±60° | ID: 0.640 -> 0.706 | Helped |
| Add an occlusion-aware triplane discriminator (ablation) | PanoHead inversion · FFHQ + LPFF | FID: 89.50 -> 64.02 | Helped |
| Add an occlusion-aware triplane discriminator (ablation) | PanoHead inversion · FFHQ + LPFF | ID: 0.87 -> 0.75 | Helped |
| Use an image-domain discriminator instead of a triplane occlusion-aware discriminator (ablation) | PanoHead inversion · FFHQ + LPFF | LPIPS: 0.14 -> 0.17 | Hurt |
| Remove triplane occlusion masking (ablation) | triplane discriminator · FFHQ + LPFF | FID: 64.02 -> 66.24 | Hurt |
| Project into Z rather than W+ (ablation) | PanoHead inversion encoder · FFHQ + LPFF | ID: 0.86 -> 0.31 | Hurt |
| Project into Z+ rather than W+ (ablation) | PanoHead inversion encoder · FFHQ + LPFF | ID: 0.86 -> 0.60 | Hurt |
| Train with synthetic Z+ images only rather than real images (ablation) | W+ encoder · FFHQ + LPFF | ID: 0.86 -> 0.25 | Hurt |
Surprising findings
- Using no discriminator gives strong same-view reconstruction but very poor novel-view FID (89.50), and an image-domain discriminator hurts input fidelity (ID 0.67).
Limitations
- Struggles with out-of-domain or tail samples such as hats and accessories, since projection is constrained to PanoHead's latent space.
- Visible artifacts in back-middle of head and mesh smoothness remain imperfect.
- Fidelity to input and 3D consistency of novel views have room for improvement.
Experimental setup
Datasets: FFHQ LPFF CelebA-HQ MEAD
Benchmarks: FFHQ + LPFF validation CelebA-HQ MEAD multi-view
Metrics: L2 LPIPS ID FID inference time
Baselines: W+ optimization PTI pSp e4e TriplaneNetv2 GOAE
Builds on: PanoHead pSp EG3D WGAN Ranger
Closest papers by meaning
- In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing (CVPR 2024 · similarity 0.89)
- WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting (NeurIPS 2025 · similarity 0.89)
- CGS-GAN: 3D Consistent Gaussian Splatting GANs for High Resolution Human Head Synthesis (NeurIPS 2025 · similarity 0.87)
- What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs (CVPR 2024 · similarity 0.86)
- G-NeRF: Geometry-enhanced Novel View Synthesis from Single-View Images (CVPR 2024 · similarity 0.86)
- Reference-Based 3D-Aware Image Editing with Triplanes (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.