Research map / 3D vision / NeRF and radiance fields
Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gear-NeRF extends dynamic NeRFs with SAM-derived 4D semantic embeddings to assign motion-based "gear" levels that adapt spatio-temporal sampling, enabling higher-quality dynamic novel view synthesis and click-based free-viewpoint object tracking.
Problem
Dynamic NeRFs lose reconstruction quality under limited compute budgets and lack semantic understanding of scenes, with uniform spatio-temporal sampling ignoring differing motion scales across regions.
Approach
Stratify dynamic regions into gear levels by motion scale using a 4D SAM-distilled semantic embedding and SAM-guided upshift masks, then allocate higher temporal feature resolution and denser ray sampling to higher-gear regions; the same embedding supports click-based free-viewpoint tracking.
Key results
- On Technicolor, Gear-NeRF reaches 32.21 PSNR and 0.058 LPIPS, versus 31.04 PSNR and 0.092 LPIPS for HyperReel.
- On Google Immersive, Gear-NeRF achieves 28.74 PSNR versus 28.32 for HyperReel; on Neural 3D Video it achieves 31.80 PSNR versus 31.12 for HyperReel.
- Free-viewpoint tracking reaches 97.4 mIoU on Technicolor versus 96.4 for SA3D, and 92.1 t-mIoU at novel time steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Gear-NeRF instead of HyperReel | dynamic novel view synthesis · Technicolor | PSNR: 31.04 -> 32.21 | Helped |
| use Gear-NeRF instead of HyperReel | dynamic novel view synthesis · Google Immersive | PSNR: 28.32 -> 28.74 | Helped |
| use Gear-NeRF instead of HyperReel | dynamic novel view synthesis · Neural 3D Video | PSNR: 31.12 -> 31.80 | Helped |
| use uniform medium temporal resolution (25) instead of motion-aware temporal sampling (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 26.93 | Hurt |
| use uniform dense temporal resolution (100) instead of motion-aware temporal sampling (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 26.85 | Hurt |
| replace motion-aware spatial sampling with uniform sampling (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 24.80 | Hurt |
| replace motion-aware spatial sampling with sample prediction network sampling (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 26.54 | Hurt |
| remove SAM embedding (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 27.10 | Hurt |
| use 2 gears instead of 4 (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 26.93 | Hurt |
| use 3 gears instead of 4 (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 27.14 | Hurt |
| use 5 gears instead of 4 (ablation) | Gear-NeRF · Truck | PSNR: 27.49 -> 27.46 | No clear effect |
Surprising findings
- On Neural 3D Video, Gear-NeRF has lower SSIM (0.936) than HyperReel (0.928 is lower; MixVoxels reports 0.944), so it does not win on every metric in that table.
Limitations
- Longer training time than several baselines (e.g., MixVoxels trains in 14-19 min versus 148-204 min for Gear-NeRF).
- Tracking at novel time steps is lower than at novel views (t-mIoU 90.6-92.1 vs mIoU 93.4-97.4).
- Ablations are run only on a single scene (Truck from Google Immersive).
Experimental setup
Datasets: Technicolor Neural 3D Video Google Immersive
Benchmarks: Technicolor Neural 3D Video Google Immersive
Metrics: PSNR SSIM LPIPS mIoU Acc. t-mIoU t-Acc.
Baselines: ST-NeRF HexPlane HyperReel MixVoxels SA3D
Builds on: NeRF HyperReel Segment Anything Model SA3D planar-factorized 4D volumes
Closest papers by meaning
- DynaMoDe-NeRF: Motion-aware Deblurring Neural Radiance Field for Dynamic Scenes (CVPR 2025 · similarity 0.88)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.88)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.88)
- Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting (ICLR 2024 · similarity 0.87)
- GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding (CVPR 2024 · similarity 0.87)
- TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.