Research map / 3D vision / Gaussian splatting
GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GaussianEditor, a framework that edits 3D Gaussian Splatting scenes with text instructions by extracting a text RoI with an LLM, grounding it to a 3D Gaussian RoI, and running InstructPix2Pix-guided edits restricted to that RoI.
Problem
Text-instruction 3D scene editing with 2D diffusion models changes the whole scene and cannot localize edits precisely, and NeRF-based representations couple spatial positions, making delicate local editing hard.
Approach
Exploit the explicit, individually-addressable nature of 3D Gaussians: obtain an LLM-extracted text RoI, ground it via Grounding DINO + SAM, lift the 2D RoI to a per-Gaussian RoI attribute trained with an SA3D-style loss, and back-propagate diffusion editing gradients only into Gaussians inside that RoI.
Key results
- On the bicycle scene, GaussianEditor (3DGS) reaches CTIDS 0.28, IIS 0.95, FID 51 in 20 min, versus IN2N at 0.22, 0.85, 103 in 51 min.
- User study with 21 participants gave GaussianEditor 87.07% of votes versus 12.93% for IN2N.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GaussianEditor with 3DGS rather than IN2N | 3D scene editing on Mip-NeRF360 bicycle · Mip-NeRF360 bicycle | CTIDS: 0.22 -> 0.28 | Helped |
| Use GaussianEditor with 3DGS rather than IN2N | 3D scene editing on Mip-NeRF360 bicycle · Mip-NeRF360 bicycle | IIS: 0.85 -> 0.95 | Helped |
| Use GaussianEditor with 3DGS rather than IN2N | 3D scene editing on Mip-NeRF360 bicycle · Mip-NeRF360 bicycle | FID: 103 -> 51 | Helped |
| Use GaussianEditor with 3DGS rather than IN2N | 3D scene editing on Mip-NeRF360 bicycle · Mip-NeRF360 bicycle | training time: 51 min -> 20 min | Helped |
| Add Gaussian RoI gradient control (ablation) | GaussianEditor · Doll, “Turn its mouth into red” | editing precision | Helped |
| Add LLM-based Text RoI extraction (ablation) | GaussianEditor · Doll, “Turn its mouth into red” | editing precision | Helped |
| Add RoI lifting from image space to 3D Gaussians (ablation) | GaussianEditor · Doll, “Turn its mouth into red” | editing precision | Helped |
| Add scene description generation (ablation) | GaussianEditor · Park, “Turn the thing next to the bike orange” | editing accuracy | Helped |
| Use multi-view scene description instead of a randomly sampled single-view description (ablation) | GaussianEditor · Park, “Turn the thing next to the bike orange” | editing accuracy | Helped |
Limitations
- Inconsistent per-view descriptions can lead the LLM to treat one object as multiple objects.
- Fails when grounding segmentation or the diffusion model completely fails, such as drastic geometric editing.
- Grounding segmentation can produce noisy image RoIs, requiring user interaction (Gaussian selection, 3D boxes) to correct.
Experimental setup
Datasets: Mip-NeRF360 bicycle scene
Benchmarks: Mip-NeRF360 bicycle scene
Metrics: CLIP Text-Image Direction Similarity Image-Image Similarity FID training time
Baselines: Instruct-NeRF2NeRF Ours-DVGO
Builds on: 3D Gaussian Splatting Instruct-NeRF2NeRF InstructPix2Pix SA3D
Closest papers by meaning
- GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (CVPR 2024 · similarity 0.94)
- 3D Gaussian Splatting as Markov Chain Monte Carlo (NeurIPS 2024 · similarity 0.88)
- 3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting (ICLR 2025 · similarity 0.88)
- D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup (NeurIPS 2024 · similarity 0.88)
- Hierarchical Gaussian Mixture Model Splatting for Efficient and Part Controllable 3D Generation (CVPR 2025 · similarity 0.87)
- GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.