Research map / 3D vision / 3D generation
WarpGAN: Warping-Guided 3D GAN Inversion with Style-Based Novel View Inpainting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WarpGAN, a 3D GAN inversion method that warps the input image to novel views using a depth map from EG3D and inpaints occluded regions with a style-modulated, symmetry-aware inpainting network (SVINet).
Problem
Single-image 3D GAN inversion for novel view synthesis yields poor-quality occluded regions because encoder-based methods rely only on the 3D GAN generative prior, which loses detail in the low bit-rate latent code.
Approach
Combines a warping-and-inpainting strategy with 3D GAN inversion: depth-based warping of the input view to novel views, then a LaMa-based inpainting network modulated by the latent code w+ and fed with mirrored-image (facial symmetry) features; training uses re-warping back to the source view so the input image serves as supervision.
Key results
- On CelebA-HQ, WarpGAN achieves FID 19.12 and ID 0.7882, beating HFGI3D (FID 24.30) and other baselines.
- On MEAD ±60°, WarpGAN reports LPIPS 0.3008 and FID 64.01, the best among the compared methods.
- Inference takes 0.0839 s on an RTX 4090, comparable to encoder-based methods and far faster than HFGI3D (264.5 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WarpGAN instead of HFGI3D | 3D GAN inversion · CelebA-HQ | FID: 24.30 -> 19.12 | Helped |
| Use WarpGAN instead of Triplanenet | 3D GAN inversion · CelebA-HQ | ID similarity: 0.7706 -> 0.7882 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±30° | LPIPS: 0.2771 -> 0.2490 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±60° | LPIPS: 0.3341 -> 0.3008 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±30° | FID: 39.30 -> 38.15 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±60° | FID: 64.47 -> 64.01 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±30° | ID similarity: 0.8109 -> 0.8315 | Helped |
| Use WarpGAN instead of the strongest baseline | 3D GAN inversion · MEAD ±60° | ID similarity: 0.7582 -> 0.7741 | Helped |
| Add SVINet to the inversion encoder (ablation) | WarpGAN · CelebA-HQ | FID: 29.28 -> 19.12 | Helped |
| Add latent-code convolution modulation (ablation) | WarpGAN · CelebA-HQ | FID: 19.47 -> 19.12 | Helped |
| Add the facial symmetry prior (ablation) | WarpGAN · CelebA-HQ | FID: 20.04 -> 19.12 | Helped |
| Add synthetic training data (ablation) | WarpGAN · CelebA-HQ | FID: 19.18 -> 19.12 | Helped |
Surprising findings
- Removing the latent-code modulation and consistency loss changes FID only modestly (19.71 vs 19.12), and training with synthetic data gives only a slight gain (19.18 vs 19.12).
Limitations
- Depth-map errors make warped images unreliable, and SVINet cannot fully remove the resulting artifacts.
- Quality degrades for large viewpoint changes because the method relies on visible regions to inpaint occluded areas.
- Evaluated mainly on face datasets; the cat domain is only a qualitative demonstration.
Experimental setup
Datasets: FFHQ CelebA-HQ MEAD AFHQ-CAT synthetic EG3D pairs (100K)
Benchmarks: CelebA-HQ MEAD
Metrics: FID ID similarity LPIPS inference time
Baselines: SG2 W+ PTI Pose Opt. HFGI3D pSp GOAE Triplanenet Dual Encoder
Builds on: EG3D LaMa GOAE PTI HFGI3D
Closest papers by meaning
- 3D GAN Inversion With Facial Symmetry Prior (CVPR 2023 · similarity 0.92)
- High-Fidelity 3D GAN Inversion by Pseudo-Multi-View Optimization (CVPR 2023 · similarity 0.92)
- In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing (CVPR 2024 · similarity 0.89)
- Dual Encoder GAN Inversion for High-Fidelity 3D Head Reconstruction from Single Images (NeurIPS 2024 · similarity 0.89)
- Spatial-Contextual Discrepancy Information Compensation for GAN Inversion (AAAI 2024 · similarity 0.86)
- What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.