Research map / 3D vision / Gaussian splatting
Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instruct-4DGS edits a 4D Gaussian Splatting dynamic scene by editing only the static canonical 3D Gaussians using InstructPix2Pix on first-timestep multiview images, then refines temporal artifacts with score distillation using Coherent-IP2P.
Problem
Instruction-guided 4D dynamic scene editing with existing methods requires editing thousands of 2D images and retraining the whole scene, taking hours and scaling poorly with the number of timesteps.
Approach
Exploit the static-dynamic separability of 4D Gaussian Splatting (canonical 3D Gaussians plus Hexplane deformation field): edit only the canonical static Gaussians from a few first-timestep multiview images, then correct temporal misalignment via score distillation with a cross-attention (Coherent) InstructPix2Pix instead of editing all frames.
Key results
- Instruct-4DGS achieves average CLIP similarity 0.249 vs 0.230 for Instruct 4D-to-4D and lower LPIPS (0.303 vs 0.491) on DyNeRF, though PSNR is lower (19.25 vs 20.40).
- Average editing time drops from about 2 hours on 2 GPUs to about 40 minutes on 1 GPU.
- Ablation user study (50 participants) prefers the full method over variants with fully SDS editing, original IP2P refinement, or refining the deformation field.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Instruct-4DGS instead of Instruct 4D-to-4D | Dynamic scene editing · DyNeRF (Average) | PSNR: 20.40 -> 19.25 | Hurt |
| Use Instruct-4DGS instead of Instruct 4D-to-4D | Dynamic scene editing · DyNeRF (Average) | SSIM: 0.736 -> 0.783 | Helped |
| Use Instruct-4DGS instead of Instruct 4D-to-4D | Dynamic scene editing · DyNeRF (Average) | LPIPS VGG: 0.491 -> 0.303 | Helped |
| Use Instruct-4DGS instead of Instruct 4D-to-4D | Dynamic scene editing · DyNeRF (Average) | CLIP similarity: 0.230 -> 0.249 | Helped |
| Use Instruct-4DGS instead of Instruct 4D-to-4D | Dynamic scene editing · Dynamic scene editing | editing time: 2 hours -> 40 minutes | Helped |
| Add 3D Gaussian editing before score-based temporal refinement (ablation) | Instruct-4DGS · Dynamic scene editing | instruction alignment and visual fidelity | Helped |
| Use Coherent-IP2P instead of original IP2P for refinement (ablation) | Instruct-4DGS · Dynamic scene editing | visual quality | Helped |
| Refine the deformation field {E, D} (ablation) | Instruct-4DGS · Dynamic scene editing | temporal consistency | Hurt |
Surprising findings
- Refining the deformation field introduces temporal inconsistencies and motion artifacts, so refining only the static Gaussians works better.
- Lower PSNR and SSIM than the baseline in several cases, attributed to the efficient editing strategy that does not directly fit all edited images.
Limitations
- Relies on InstructPix2Pix capabilities
- Cannot directly edit motion and requires segmentation for partial edits
- Motion artifacts remain due to 4D representation limits even after refinement
Experimental setup
Datasets: DyNeRF Technicolor
Benchmarks: DyNeRF Technicolor
Metrics: PSNR SSIM LPIPS CLIP similarity editing time user preference
Baselines: Instruct 4D-to-4D
Builds on: 4D Gaussian Splatting InstructPix2Pix DreamFusion score distillation Instruct 4D-to-4D
Closest papers by meaning
- Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion (CVPR 2024 · similarity 0.91)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.90)
- Control4D: Efficient 4D Portrait Editing with Text (CVPR 2024 · similarity 0.90)
- D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup (NeurIPS 2024 · similarity 0.90)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.89)
- 1000+ FPS 4D Gaussian Splatting for Dynamic Scene Rendering (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.