Research map / Image generation / Editing and style transfer
Control4D: Efficient 4D Portrait Editing with Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Control4D edits dynamic 4D human portraits from multi-view video using text prompts by combining a plane-decomposed Gaussian Splatting representation (GaussianPlanes) with a GAN-based 4D generator trained on ControlNet-edited images.
Problem
Text-driven 4D portrait editing suffers from slow, memory-heavy dynamic NeRF representations and from spatial-temporal inconsistency in frame-by-frame diffusion-based edits.
Approach
Structure Gaussian Splatting with tri-plane features for attributes and 9-plane (Tensor4D-style) decomposition for flow, then learn a continuous GAN generation space from inconsistent ControlNet edits via a discriminator and multi-level (global/local encoder) guidance, instead of direct supervision.
Key results
- On the dynamic scene, Control4D reaches FID 18.59 and CLIP similarity 0.3192, versus 47.39 / 0.3178 for Tensor4D+GAN and 155.6 / 0.3144 for Tensor4D.
- On static scenes, Control4D achieves FID 14.11 versus 126.3 for InstructNeRF2NeRF (CLIP 0.3323 vs 0.2989).
- Multi-level guidance ablation: removing levels 2 and 3 yields FID 45.19 (only levels 1 and 3) or 54.29 (only level 3) versus 18.59 for the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Control4D instead of GaussianPlanes alone | GaussianPlanes · Dynamic scene | FID: 67.58 -> 18.59 | Helped |
| Add GAN-based generation to Tensor4D | Tensor4D · Dynamic scene | FID: 155.6 -> 47.39 | Helped |
| Use Control4D instead of Tensor4D+GAN | Tensor4D+GAN · Dynamic scene | FID: 47.39 -> 18.59 | Helped |
| Use Control4D instead of InstructNeRF2NeRF | InstructNeRF2NeRF · Static scene | FID: 126.3 -> 14.11 | Helped |
| Omit the 2nd level of multi-level guidance (ablation) | Control4D · Tensor4D dataset, dynamic scene | FID: 18.59 -> 45.19 | Hurt |
| Omit the 2nd level of multi-level guidance (ablation) | Control4D · Tensor4D dataset, dynamic scene | CLIP Similarity: 0.3192 -> 0.3174 | Hurt |
| Omit the 1st and 2nd levels of multi-level guidance (ablation) | Control4D · Tensor4D dataset, dynamic scene | FID: 18.59 -> 54.29 | Hurt |
| Omit the 1st and 2nd levels of multi-level guidance (ablation) | Control4D · Tensor4D dataset, dynamic scene | CLIP Similarity: 0.3192 -> 0.3151 | Hurt |
Surprising findings
- Plain ControlNet editing reached a higher CLIP similarity than InstructPix2Pix, and InstructNeRF2NeRF scored lower CLIP similarity than the Tensor4D variants.
Limitations
- Relies on canonical Gaussian flow, struggling with rapid or large non-rigid motion.
- Editing granularity is coarse because it is bounded by ControlNet; precise expression or action edits are not possible.
- Requires iterative optimization and cannot edit in a single step.
Experimental setup
Datasets: Tensor4D Twindom
Benchmarks: Tensor4D Twindom
Metrics: FID CLIP similarity
Baselines: InstructPix2Pix ControlNet InstructNeRF2NeRF Tensor4D Tensor4D+GAN GaussianPlanes
Builds on: Gaussian Splatting Tensor4D Instruct-NeRF2NeRF ControlNet Wasserstein GAN gradient penalty
Closest papers by meaning
- Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion (CVPR 2024 · similarity 0.91)
- Efficient Dynamic Scene Editing via 4D Gaussian-based Static-Dynamic Separation (CVPR 2025 · similarity 0.90)
- CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusion (CVPR 2025 · similarity 0.88)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.87)
- GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (CVPR 2024 · similarity 0.87)
- A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.