Research map / Video / Video generation
DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DynVideo-E, which edits human-centric videos by editing a dynamic NeRF (HOSNeRF-based video-NeRF) with reference-image reconstruction, multi-view multi-pose SDS from personalized 2D and Zero-1-to-3 3D priors, local part super-resolution, and style transfer on the background.
Problem
Diffusion-based video editing is limited to short videos with small motion and viewpoint changes; existing 2D video representations (neural atlases, canonical images) fail on large-scale motion and view changes in human-centric videos.
Approach
Use a dynamic NeRF (human pose-guided deformation field plus a 3D background model) as the video representation so edits are made in 3D space and propagated to all frames via the deformation field, with image-based reference-subject and style guidance distilled through SDS from 2D personalized and 3D diffusion priors.
Key results
- DynVideo-E achieves CLIPScore 31.31, highest among all methods (best baseline Text2Video-Zero 26.70).
- Human preference outperforms all baselines by about 50%-95% margin across textual faithfulness, temporal consistency, and overall quality (1140 comparisons, 32 raters on MTurk).
- Ablation shows CLIP image similarity drops from 0.756 (full) to 0.698 on Backpack when removing super-resolution, reconstruction, 3D SDS, and 2D LoRA components.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DynVideo-E instead of Text2Video-Zero | human-centric video editing · HOSNeRF and NeuMan datasets | human preference (textual faithfulness): 9.17 v.s. 90.83 (Ours) | Helped |
| Use DynVideo-E instead of Rerender-A-Video | human-centric video editing · HOSNeRF and NeuMan datasets | human preference (overall quality): 9.58 v.s. 90.42 (Ours) | Helped |
| Use DynVideo-E instead of Text2LIVE | human-centric video editing · HOSNeRF and NeuMan datasets | human preference (overall quality): 9.05 v.s. 90.95 (Ours) | Helped |
| Use DynVideo-E instead of StableVideo | human-centric video editing · HOSNeRF and NeuMan datasets | human preference (overall quality): 6.19 v.s. 93.81 (Ours) | Helped |
| Use DynVideo-E instead of CoDeF | human-centric video editing · HOSNeRF and NeuMan datasets | human preference (overall quality): 1.25 v.s. 98.75 (Ours) | Helped |
| Add local parts super-resolution (ablation) | DynVideo-E · Backpack scene | CLIP image cosine similarity: 0.736 -> 0.756 | Helped |
| Add local parts super-resolution (ablation) | DynVideo-E · Lab scene | CLIP image cosine similarity: 0.645 -> 0.647 | Helped |
| Add reconstruction loss (ablation) | DynVideo-E · Backpack scene | CLIP image cosine similarity: 0.728 -> 0.736 | Helped |
| Add reconstruction loss (ablation) | DynVideo-E · Lab scene | CLIP image cosine similarity: 0.617 -> 0.645 | Helped |
| Add 2D SDS (ablation) | DynVideo-E · Backpack scene | CLIP image cosine similarity: 0.679 -> 0.728 | Helped |
| Add 2D SDS (ablation) | DynVideo-E · Lab scene | CLIP image cosine similarity: 0.517 -> 0.617 | Helped |
| Add 3D SDS (ablation) | DynVideo-E · Backpack scene | CLIP image cosine similarity: 0.711 -> 0.728 | Helped |
Surprising findings
- Baselines such as CoDeF, Text2LIVE, and StableVideo overfit to input frames on large-motion videos and learn meaningless canonical images or neural atlases.
Limitations
- NeRF-based representation is time-consuming to train and edit.
- Training runs at 128x128 resolution due to GPU memory, giving coarse geometry and blurry textures before super-resolution.
- Evaluation uses a small set (24 prompts, 11 videos) and crowd-sourced human preference with 32 raters.
Experimental setup
Datasets: HOSNeRF dataset NeuMan dataset
Benchmarks: HOSNeRF dataset NeuMan dataset
Metrics: CLIPScore human preference (textual faithfulness, temporal consistency CLIP image similarity
Baselines: Text2Video-Zero Rerender-A-Video Text2LIVE StableVideo CoDeF
Builds on: HOSNeRF Zero-1-to-3 Magic123 DreamHuman ARF Mip-NeRF 360
Closest papers by meaning
- Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos (AAAI 2024 · similarity 0.88)
- DynaMoDe-NeRF: Motion-aware Deblurring Neural Radiance Field for Dynamic Scenes (CVPR 2025 · similarity 0.87)
- ED-NeRF: Efficient Text-Guided Editing of 3D Scene With Latent Space NeRF (ICLR 2024 · similarity 0.87)
- NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing (NeurIPS 2024 · similarity 0.86)
- VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling (NeurIPS 2026 · similarity 0.86)
- TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.