Research map / 3D vision / NeRF and radiance fields
Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds learnable per-camera time offsets to dynamic NeRFs so unsynchronized multi-view videos can be fit, and the offsets double as automatic video synchronization.
Problem
Dynamic NeRFs assign one latent embedding per frame index, so multi-view videos that are not temporally synchronized cannot be fit, even on training views.
Approach
Learn a time offset per camera, jointly with the NeRF, so each view's frame index is shifted to its true capture moment; additionally use an implicit MLP over sinusoidally encoded time to produce continuous temporal embeddings, and bilinear interpolation for grid-based models.
Key results
- On the unsynchronized Plenoptic Video Dataset test views, Sync-MixVoxels improves average PSNR from 29.96 to 30.53 and Sync-K-Planes improves from 29.16 to 30.44.
- On the Unsynchronized Dynamic Blender Dataset, Sync-K-Planes raises average test PSNR from 32.66 to 39.40 and Sync-MixVoxels from 31.06 to 37.11.
- Mean absolute error of learned time offsets on the synthetic dataset is about 0.01 s (0.0154 to 0.0229 s), under one-third of a frame.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable per-camera time offsets and continuous temporal embeddings | MixVoxels · Unsynchronized Plenoptic Video Dataset test view | PSNR: 29.96 -> 30.53 | Helped |
| Add learnable per-camera time offsets | K-Planes · Unsynchronized Plenoptic Video Dataset test view | PSNR: 29.16 -> 30.44 | Helped |
| Add learnable per-camera time offsets and continuous temporal embeddings | MixVoxels · Unsynchronized Dynamic Blender Dataset test view | PSNR: 31.06 -> 37.11 | Helped |
| Add learnable per-camera time offsets | K-Planes · Unsynchronized Dynamic Blender Dataset test view | PSNR: 32.66 -> 39.40 | Helped |
| Optimize the test-view time offset (ablation) | Sync-MixVoxels · Unsynchronized Plenoptic Video Dataset test view | PSNR: 30.29 -> 30.53 | Helped |
| Optimize the test-view time offset (ablation) | Sync-K-Planes · Unsynchronized Plenoptic Video Dataset test view | PSNR: 30.25 -> 30.44 | Helped |
| Use sinusoidal time encoding with L=10 instead of L=0 (ablation) | Sync-MixVoxels · flame_salmon | PSNR: 22.26 -> 28.85 | Helped |
| Use sinusoidal time encoding with L=10 instead of L=5 (ablation) | Sync-MixVoxels · flame_salmon | PSNR: 28.80 -> 28.85 | Helped |
| Increase K-Planes temporal-axis resolution (ablation) | K-Planes · flame_salmon | PSNR: 26.19 -> 26.56; 25.61 -> 25.31 | Mixed |
| Increase temporal-axis resolution (ablation) | Sync-K-Planes · flame_salmon | PSNR: 29.17 -> 28.95; 28.56 -> 28.71 | Mixed |
| Add learnable per-camera time offsets | MixVoxels · Synchronized Plenoptic Video Dataset test view | PSNR: 30.39 -> 30.41 | Helped |
| Add learnable per-camera time offsets | K-Planes · Synchronized Plenoptic Video Dataset test view | PSNR: 30.40 -> 30.56 | Helped |
Surprising findings
- Baselines fail to fit even training views when videos are unsynchronized, and K-Planes degrades more than MixVoxels under unsynchronization despite outperforming it on synchronized data.
- Sync-K-Planes also gives small gains on the synchronized Plenoptic dataset, suggesting the original 'synchronized' data has sub-frame offsets.
Limitations
- Does not handle causal handheld cameras.
- Implicit-function variant needs extra training time and memory.
Experimental setup
Datasets: Plenoptic Video Dataset Unsynchronized Dynamic Blender Dataset
Benchmarks: Plenoptic Video Dataset Unsynchronized Dynamic Blender Dataset
Metrics: PSNR SSIM LPIPS (VGG) LPIPS (AlexNet) MAE of time offsets
Baselines: MixVoxels K-Planes
Builds on: NeRF MixVoxels K-Planes DyNeRF NeRF--
Closest papers by meaning
- DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing (CVPR 2024 · similarity 0.88)
- NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows (CVPR 2024 · similarity 0.88)
- Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video (CVPR 2025 · similarity 0.88)
- TFS-NeRF: Template-Free NeRF for Semantic 3D Reconstruction of Dynamic Scene (NeurIPS 2024 · similarity 0.87)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.87)
- DynaMoDe-NeRF: Motion-aware Deblurring Neural Radiance Field for Dynamic Scenes (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.