Research map / Image generation / Editing and style transfer

Reference-Based 3D-Aware Image Editing with Triplanes

CVPR 2025 · Image generation › Editing and style transfer · approach: GAN inversion for editing · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a training-light pipeline for reference-based, 3D-aware part editing of single images in EG3D triplane space, using gradient-based triplane masking, implicit fusion via encoding/decoding, and a fine-tuned W+ encoder.

Problem

Reference-based image editing that copies local attributes from a reference image into a source image while staying 3D-consistent and faithful to both images; existing methods are either 2D-only, lack reference-based control, or do not support 3D-aware local edits.

Approach

Localize facial or object parts in triplane features by backpropagating 2D segmentation masks rendered from multiple views into the triplane, stitch reference and source triplanes with the masks, then re-encode and decode the fused triplane through a W+ encoder (high-frequency branch disabled) to implicitly fuse, and fine-tune that encoder with masked LPIPS and ID losses.

Key results

• On CelebA eyeglasses edit, the method reaches FID 66.68 and MSSIM 0.9818, versus the best baseline FID 71.47 (VecGAN++) and MSSIM 0.9645 (LEDITS++).
• On CelebA hair edit, FID 64.59 vs. 62.80 for Barbershop, which is the strongest FID baseline; the method has the best ML2 of 0.0029 range among methods reporting hair results except HiSD (0.0036), and MSSIM 0.9720 close to HiSD's 0.9743.
• In a 25-person user study on FFHQ and AFHQ, participants identified edited images at roughly chance-level rates (e.g., 49% overall on FFHQ), suggesting the edits are hard to distinguish.

Measured effects reported in the paper

Surprising findings

• Even a 2D pixel-based baseline's FID comparisons show Barbershop has a lower hair FID than the proposed method, which the paper does not highlight as a win.

Limitations

• Relies on the underlying generator (EG3D, AG3D, PanoHead, LN3Diff, InstantMesh), which may struggle with background generation and high-quality reconstruction.
• Background details may not be fully preserved in some cases.
• Evaluation of the editing quality is limited to CelebA-based FID and masked metrics with a small user study.

Experimental setup

Datasets: CelebA FFHQ AFHQ DeepFashion

Benchmarks: CelebA

Metrics: FID masked SSIM (MSSIM) masked L2 (ML2) user study identification rate

Baselines: HiSD VecGAN++ Barbershop HairCLIPv2 StyleFusion InterFaceGAN StyleCLIP SFE E3DGE NoiseCLR LEDITS++ InfEdit

Builds on: EG3D Rosinality-style W+ encoder (ReStyle/ E3DGE-type inversion) InterFaceGAN LPIPS ArcFace identity loss

Closest papers by meaning

• Navigating the GAN Parameter Space for Semantic Image Editing (CVPR 2021 · similarity 0.89)
• Editing in Style: Uncovering the Local Semantics of GANs (CVPR 2020 · similarity 0.89)
• In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing (CVPR 2024 · similarity 0.87)
• E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.87)
• InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image (ICLR 2024 · similarity 0.86)
• Dual Mapping of 2D StyleGAN for 3D-Aware Image Generation and Manipulation (Student Abstract) (AAAI 2024 · similarity 0.86)