Research map / 3D vision / 3D generation
NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transforms a pretrained NeRF from a single RGBD view of a non-rigidly deformed scene by modeling a 3D scene flow as linear blending of anchor-point rigid transforms, with NeRF-based correspondence matching and a new 113-scene Objaverse dataset.
Problem
Modifying a NeRF of an original scene to match a non-rigidly transformed scene observed only through a single RGBD image, recovering geometry and novel views without user input.
Approach
Defines the forward and backward scene flow as a KNN-weighted linear blend of local rigid transforms anchored at mesh vertices extracted from the NeRF, optimized with ARAP and 3D correspondence consistency losses; correspondences come from multi-view ASpanFormer matches filtered in pixel and 3D space.
Key results
- Ours achieves 25.9 PSNR, 0.924 SSIM and 0.061 LPIPS for novel view synthesis, versus 22.1 PSNR for SINE and 19.8 PSNR for DreamGaussian.
- Ours reaches a 90.3% geometric reconstruction success rate and 0.666 VmIoU, compared with 63.7% and 0.515 for SINE.
- Replacing the proposed linear-blend flow with an MLP cyclic flow drops PSNR from 25.9 to 22.5 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NeRFDeformer rather than SINE | NeRF transformation · 113-scene dataset | PSNR: 22.1±3.8 -> 25.9±4.2 | Helped |
| use NeRFDeformer rather than SINE | NeRF transformation · 113-scene dataset | LPIPS: 0.115±0.053 -> 0.061±0.040 | Helped |
| use ASpanFormer rather than FlowFormer for matching (ablation) | NeRFDeformer with single original image and no filtering · 113-scene dataset | PSNR: 22.6±4.4 -> 23.9±4.2 | Helped |
| use multiple original images rather than a single original image (ablation) | NeRFDeformer correspondence matching · 113-scene dataset | PSNR: 23.9±4.2 -> 25.9±4.2 | Helped |
| use the proposed 2D+3D correspondence filtering rather than naive 2D filtering (ablation) | NeRFDeformer with ASpanFormer and multiple original images · 113-scene dataset | PSNR: 22.1±4.0 -> 25.9±4.2 | Helped |
| use the proposed scene flow rather than MLP flow (ablation) | NeRFDeformer with ASpanFormer, multiple images, and 2D+3D filtering · 113-scene dataset | PSNR: 22.5±3.5 -> 25.9±4.2 | Helped |
| add SimKinect depth noise at s.d. 0.3, 0.5, or 1.0 (ablation) | NeRFDeformer · 113-scene dataset | PSNR: 23.6 to 24.4 | Hurt |
Surprising findings
- Fine-tuning the original NeRF on the single transformed view causes the NeRF to collapse (success rate 19.5%, CD 228).
- Diffusion-based Zero123-XL underperforms even the unmodified original NeRF, since its priors may not cover the scene content.
Limitations
- Requires a depth input (RGBD) for the transformed view; the authors note relaxing this is future work.
- Success rate is 90%, so a meaningful fraction of scenes fail, and the flow is only defined near the surface.
- Evaluated only on synthetic Objaverse scenes plus one real-world experiment.
Experimental setup
Datasets: Objaverse
Benchmarks: NeRFDeformer dataset (113 scenes)
Metrics: PSNR SSIM LPIPS chamfer distance volume IoU success rate
Baselines: Zero123-XL DreamGaussian NeRF finetuned SINE Original NeRF
Builds on: Instant-NGP ASpanFormer embedded deformation graphs SINE Zero123-XL
Closest papers by meaning
- NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs (CVPR 2024 · similarity 0.90)
- MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior (CVPR 2024 · similarity 0.90)
- TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene (NeurIPS 2024 · similarity 0.88)
- Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos (AAAI 2024 · similarity 0.88)
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods (NeurIPS 2025 · similarity 0.87)
- SANeRF-HQ: Segment Anything for NeRF in High Quality (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.