Research map / 3D vision / 3D generation
SHAP-EDITOR: Instruction-Guided Latent 3D Editing in Seconds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SHAP-EDITOR, a feed-forward latent 3D editor that is initialised from Shap-E and distilled from 2D text-to-image and instruction-based image editors, enabling text-driven 3D object edits in about one second.
Problem
Text-driven 3D editing is typically done by per-instance test-time optimisation (distillation from 2D models), taking minutes to hours per asset, which is impractical for interactive or large-scale use.
Approach
Learn a single feed-forward latent editor operating in the Shap-E 3D latent space (initialised from Shap-E's diffusion transformer) and supervise it with SDS losses from InstructPix2Pix/MagicBrush (TI2I) and Stable Diffusion (T2I), plus depth/photometric regularisers and cross-attention masks for local edits.
Key results
- Single-prompt SHAP-EDITOR reaches CLIPsim 0.292 and CLIPdir 0.097 on local editing, beating test-time optimisation (0.290 / 0.087) and Vox-E (0.277 / 0.075).
- On global editing, SHAP-EDITOR gets Structure Distance 0.008 versus 0.095 for Instruct-NeRF2NeRF and 0.026 for Vox-E, with inference around one second versus tens of minutes for baselines.
- Depth regularisation is critical for global edits: removing it raises Structure Distance from 0.008 to 0.138.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the feed-forward single-prompt SHAP-EDITOR instead of Vox-E | SHAP-EDITOR · Local editing | CLIPsim: 0.277 -> 0.292 | Helped |
| Use the feed-forward single-prompt SHAP-EDITOR instead of Vox-E | SHAP-EDITOR · Global editing | Structure Dist.: 0.026 -> 0.008 | Helped |
| Add global depth regularization (ablation) | SHAP-EDITOR · Global editing | Structure Dist.: 0.138 -> 0.008 | Helped |
| Add cross-attention masks (ablation) | SHAP-EDITOR · Local editing | CLIPsim: 0.261 -> 0.292 | Helped |
| Add local regularization loss (ablation) | SHAP-EDITOR · Local editing | CLIPdir: 0.092 -> 0.097 | Helped |
| Add T2I SDS loss (ablation) | SHAP-EDITOR · Local editing | CLIPsim: 0.263 -> 0.292 | Helped |
| Add TI2I SDS loss (ablation) | SHAP-EDITOR · Local editing | CLIPsim: 0.278 -> 0.292 | Helped |
Surprising findings
- The learned feed-forward editor outperforms its own test-time optimisation counterpart using the same prompt and losses, which the authors attribute to regularisation from learning an editor.
Limitations
- Performance is upper-bounded by the quality of the Shap-E latent space and the pre-trained 2D editors.
- Not yet a fully open-ended editor; each edit type requires its own trained editor.
- Requires substantial compute for open-ended training (authors estimate hundreds of GPUs).
Experimental setup
Datasets: OmniObject3D Shap-E generated 3D objects
Benchmarks: custom 20 instance-instruction evaluation set (15 unseen 3D objects, 5 instructions)
Metrics: CLIPsim CLIPdir Structure Distance inference time
Baselines: Instruct-NeRF2NeRF Vox-E Text2Mesh Ours (Test-time Optimisation)
Builds on: Shap-E Score Distillation Sampling InstructPix2Pix MagicBrush Instruct-NeRF2NeRF
Closest papers by meaning
- PrEditor3D: Fast and Precise 3D Shape Editing (CVPR 2025 · similarity 0.87)
- ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing (NeurIPS 2024 · similarity 0.87)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.87)
- EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting (CVPR 2025 · similarity 0.87)
- DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation (ICLR 2025 · similarity 0.86)
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.