Research map / 3D vision / 3D generation
TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TFS-NeRF, a template-free semantic NeRF that uses an invertible neural network to predict forward LBS skinning for two interacting entities from sparse or single-view RGB videos, avoiding Broyden root-finding for faster training.
Problem
Reconstructing semantically separable 3D geometry of dynamic scenes with two interacting rigid, non-rigid, or deformable entities from sparse or single-view RGB videos without templates, depth, or optical flow, while keeping training time low.
Approach
Replaces the iterative Broyden root-finding used to invert forward LBS with an invertible neural network that maps view-space points to canonical space, conditioned on per-frame skeletons. It uses semantic-aware ray sampling, separate SDF networks per entity, and per-entity skinning weights with pose, skinning, cycle, and consistency losses.
Key results
- On BEHAVE scene reconstruction, the method reaches F-score 91.34% and Chamfer 2.43 cm, versus 80.39% F-score for the best ResField-trained baseline (NDR).
- On ZJU-MoCap human reconstruction, F-score 93.97% vs 92.95% for TAVA, and Chamfer 2.24 cm vs 2.47 cm.
- Training takes about 0.3 sec/frame with the INN versus 0.9 sec/frame with the Broyden-based approach.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TFS-NeRF instead of the strongest listed baseline | BEHAVE human reconstruction · BEHAVE human reconstruction | F-score: 79.000 -> 95.343 | Helped |
| use TFS-NeRF instead of the strongest listed baseline | BEHAVE object reconstruction · BEHAVE object reconstruction | F-score: 80.174 -> 86.991 | Helped |
| use TFS-NeRF instead of the strongest listed baseline | BEHAVE scene reconstruction · BEHAVE scene reconstruction | F-score: 80.385 -> 91.343 | Helped |
| use TFS-NeRF instead of NDR with ResField | HO3D-V3 hand reconstruction · HO3D-V3 hand reconstruction | F-score: 94.051 -> 95.396 | Helped |
| use TFS-NeRF instead of HyperNeRF with ResField | HO3D-V3 object reconstruction · HO3D-V3 object reconstruction | F-score: 97.988 -> 99.980 | Helped |
| use TFS-NeRF instead of TAVA | ZJU-MoCap human reconstruction · ZJU-MoCap human reconstruction | F-score: 92.95 -> 93.97 | Helped |
| use TFS-NeRF instead of TAVA | synthetic animal reconstruction · Synthetic animal reconstruction | F-score: 99.83 -> 99.99 | Helped |
| add skeleton-based canonical point initialization (ablation) | TFS-NeRF · BEHAVE scene reconstruction | F-score: 83.50 -> 91.34 | Helped |
| use separate geometry heads instead of shared heads (ablation) | TFS-NeRF · BEHAVE scene reconstruction | F-score: 41.18 -> 91.34 | Helped |
| add skinning weight loss (ablation) | TFS-NeRF · BEHAVE scene reconstruction | F-score: 82.67 -> 91.34 | Helped |
| add pose loss (ablation) | TFS-NeRF · BEHAVE scene reconstruction | F-score: 90.17 -> 91.34 | Helped |
| replace Broyden-based LBS solving with INN | TFS-NeRF · training | time per frame: 0.9 sec/frame -> 0.3 sec/frame | Helped |
Surprising findings
- On the synthetic animal dataset the method only shows comparable performance to TAVA (F-score 99.99 vs 99.83), not a clear gain.
- Removing the pose loss gives nearly the same F-score as the full model (90.17 vs 91.34) on BEHAVE scene reconstruction.
Limitations
- Uses separate networks per entity, which does not scale beyond two entities.
- Requires good-quality 3D skeletons; predicted poses degrade reconstruction quality.
- Relies on 2D semantic masks and dataset-provided camera and body poses for training.
Experimental setup
Datasets: BEHAVE HO3D-V3 ZJU-MoCap synthetic animal dataset from TAVA YCB-Video 3D models
Benchmarks: BEHAVE HO3D-V3 ZJU-MoCap synthetic animal dataset
Metrics: Dist. Acc. Completeness Precision Recall F-score Chamfer distance
Baselines: Tensor4D NDR HyperNeRF D-NeRF TAVA AnimatableNeRF ResField
Builds on: TAVA SNARF NeRF Broyden's method ResField
Closest papers by meaning
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.88)
- Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling (CVPR 2024 · similarity 0.87)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.87)
- Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos (AAAI 2024 · similarity 0.87)
- AE-NeRF: Augmenting Event-Based Neural Radiance Fields for Non-ideal Conditions and Larger Scenes (AAAI 2025 · similarity 0.87)
- REACTO: Reconstructing Articulated Objects from a Single Video (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.