Research map / 3D vision / 3D generation
InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructPix2NeRF, an end-to-end conditional latent 3D diffusion transformer that edits a single portrait from natural-language instructions into 3D-consistent NeRF/EG3D outputs, using a triplet dataset, token position randomization, and an identity consistency module.
Problem
Instructed 3D-aware portrait editing from a single open-world image is under-explored: existing 3D editing methods either lack natural-language support, need per-prompt optimization, or cannot handle real faces, and multi-instruction editing with identity preservation is hard.
Approach
Performs diffusion in the W+ latent space of EG3D (via PREIM3D encoder) conditioned on the original face latent (concatenation + adaLN identity modulation) and CLIP instruction embeddings via cross-attention, trained on a 640K triplet dataset; token position randomization enables multi-instruction editing in one pass, and an identity regularization loss on one-step predictions improves multi-view
Key results
- On single-instruction editing, InstructPix2NeRF reaches ID 0.56/0.59/0.60 for bangs/eyeglasses/smile versus best baseline ~0.51 (InstructPix2Pix) and CLIP 0.13/0.20/0.18.
- On multi-instruction editing, ours gets ID 0.55 and CLIP 0.20 with AA_min 0.52, versus 0.46/0.19 and AA_min 0.31 for InstructPix2Pix.
- User study: ours preferred over Talk-To-Edit, InstructPix2Pix and img2img at rates 0.67 to 0.96 across attributes (1,440 votes, 30 volunteers).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InstructPix2NeRF instead of InstructPix2Pix + PREIM3D | single-instruction 3D portrait editing · CelebA-HQ, bangs | ID: 0.40 -> 0.56 | Helped |
| Use InstructPix2NeRF instead of InstructPix2Pix + PREIM3D | single-instruction 3D portrait editing · CelebA-HQ, bangs | CLIP: 0.07 -> 0.13 | Helped |
| Use InstructPix2NeRF instead of InstructPix2Pix + PREIM3D | single-instruction 3D portrait editing · CelebA-HQ, eyeglasses | ID: 0.51 -> 0.59 | Helped |
| Use InstructPix2NeRF instead of InstructPix2Pix + PREIM3D | single-instruction 3D portrait editing · CelebA-HQ, smile | CLIP: 0.16 -> 0.18 | Helped |
| Add token position randomization during training (ablation) | InstructPix2NeRF · CelebA-HQ, multiple instructions | ID: 0.50 -> 0.55 | Helped |
| Add token position randomization during training (ablation) | InstructPix2NeRF · CelebA-HQ, multiple instructions | AA_min: 0.08 -> 0.52 | Helped |
| Add token position randomization during training (ablation) | InstructPix2NeRF · CelebA-HQ, multiple instructions | CLIP: 0.18 -> 0.20 | Helped |
| Add identity regularization loss (ablation) | InstructPix2NeRF · CelebA-HQ, multiple instructions | ID: 0.44 -> 0.55 | Helped |
| Add identity modulation (ablation) | InstructPix2NeRF · CelebA-HQ, multiple instructions | ID: 0.50 -> 0.55 | Helped |
| Replace the inversion encoder with direct W+ optimization at inference (ablation) | InstructPix2NeRF · 3D portrait editing | identity consistency | Hurt |
| Replace the inversion encoder with PTI optimization at inference (ablation) | InstructPix2NeRF · 3D portrait editing | identity consistency | Hurt |
| Use triplet paired-image and instruction data instead of text-image pairing (ablation) | InstructPix2NeRF · facial attribute editing | ID | Helped |
Surprising findings
- Directly replacing the encoder with W+ optimization (direct or PTI) at inference severely degrades editing and identity, attributed to a distribution mismatch with the learned model.
Limitations
- Semantically identical instructions can yield slightly different results (e.g., different pink hair tones).
- Loss of fine details such as eye shape and eyelashes.
- Training-time W+ optimization is too costly, and inference-time optimization mismatches the learned distribution.
Experimental setup
Datasets: FFHQ CelebA-HQ
Benchmarks: CelebA-HQ (first 300 images)
Metrics: ID (ArcFace similarity) directional CLIP similarity AA AD M_d S_d user study preference rate
Baselines: Talk-To-Edit + PREIM3D InstructPix2Pix + PREIM3D img2img (Stable Diffusion-style latent denoising)
Builds on: Diffusion Transformer (DiT) EG3D PREIM3D InstructPix2Pix classifier-free guidance
Closest papers by meaning
- DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis (CVPR 2024 · similarity 0.88)
- StrokeFaceNeRF: Stroke-based Facial Appearance Editing in Neural Radiance Field (CVPR 2024 · similarity 0.88)
- FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields (CVPR 2025 · similarity 0.87)
- SIGNeRF: Scene Integrated Generation for Neural Radiance Fields (CVPR 2024 · similarity 0.87)
- Reference-Based 3D-Aware Image Editing with Triplanes (CVPR 2025 · similarity 0.86)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.