Research map / 3D vision / Face and shape
What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a learned, scene-conditional proposal sampler and SDF-based triplane 3D GAN that renders every pixel of 512x512 images with about 20-30 depth samples per ray, achieving strong 3D geometry without 2D super-resolution.
Problem
Neural volume rendering in 3D GANs is too memory- and compute-intensive for native high-resolution output, so existing methods use patch-based training or low-resolution rendering with 2D super-resolution, which harms multiview consistency and geometry quality.
Approach
A CNN predicts high-resolution volume rendering proposal weights from a cheap 128x128 low-resolution probe, combined with robust nucleus-style sampling and a spatially-varying SDF softness (beta) regularization, enabling full-resolution rendering with few samples per ray.
Key results
- On FFHQ-512 without super resolution, the method reaches FID 4.97 and FID-N 60.76, the best geometry score among non-SR methods, while matching EG3D (FID 4.70) with SR.
- On AFHQv2 Cats 512, the method achieves FID 4.23 without SR, beating Mimic3D (4.29) and GRAM-HD (7.67) among neural-rendering-only methods.
- The sampler operates with about 20 samples per ray versus at least 96 for prior 3D GANs, using up to 5x fewer depth samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the learned sampler (ablation) | Ours · FFHQ-512 | FID: 38.29 -> 4.97 | Helped |
| add the learned sampler (ablation) | Ours · FFHQ-512 | FID-N: 93.88 -> 60.76 | Helped |
| add stratified sampling (ablation) | Ours · FFHQ-512 | NFS: 5.97 -> 29.35 | Helped |
| add robust sampling (ablation) | Ours · FFHQ-512 | FID: 5.67 -> 4.97 | Helped |
| add robust sampling (ablation) | Ours · FFHQ-512 | FID-N: 60.78 -> 60.76 | No clear effect |
| add beta regularization (ablation) | Ours · FFHQ-512 | FID-N: 64.25 -> 60.76 | Helped |
Surprising findings
- Removing the learned sampler causes FID to rise from 4.97 to 38.29, showing the sampler is essential for training at this sample budget.
Limitations
- Artifacts such as dents under specularities.
- Cannot handle transparent objects such as lenses well.
- Frontal bias and inaccurate camera labels cause geometry artifacts on face sides.
Experimental setup
Datasets: FFHQ AFHQv2 Cats NPHM
Benchmarks: FFHQ AFHQv2 Cats
Metrics: FID FID-N NFS
Baselines: EG3D StyleSDF MVCGAN Mimic3D Epigraf GMPI GRAM-HD
Builds on: StyleGAN2 VolSDF EG3D StyleSDF Adaptive Shells
Closest papers by meaning
- Efficient Geometry-Aware 3D Generative Adversarial Networks (CVPR 2022 · similarity 0.93)
- 3D-Aware Image Synthesis via Learning Structural and Textural Representations (CVPR 2022 · similarity 0.92)
- EpiGRAF: Rethinking training of 3D GANs (NeurIPS 2022 · similarity 0.91)
- Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator (NeurIPS 2022 · similarity 0.90)
- Dual Mapping of 2D StyleGAN for 3D-Aware Image Generation and Manipulation (Student Abstract) (AAAI 2024 · similarity 0.88)
- CGS-GAN: 3D Consistent Gaussian Splatting GANs for High Resolution Human Head Synthesis (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.