Research map / 3D vision / Depth and stereo
Visual Sync: Multi‑Camera Synchronization via Cross‑View Object Motion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisualSync, an optimization framework that synchronizes unposed, unsynchronized multi-camera videos by minimizing Sampson epipolar error over dynamic-object tracklet correspondences, using pretrained VGGT, CoTracker3, MASt3R, and DEVA.
Problem
Temporally aligning (synchronizing) multiple casually captured, unposed consumer videos of the same dynamic scene without audio, human-pose cues, or specialized hardware.
Approach
Once synchronized, moving 3D points co-visible in two cameras satisfy epipolar constraints; VisualSync exploits this by pairwise discrete offset search minimizing a Sampson epipolar error over dense cross-view tracklets, then a robust IRLS global offset solve across pairs.
Key results
- Achieves median video synchronization error of 46.6 ms on Egohumans, versus 222.1 ms for Uni4D (with GT poses) and 263.8 ms for MASt3R.
- On CMU Panoptic, median error 41.5 ms vs 58.1 ms for MASt3R; on 3D-POP, mean error 114.7 ms vs 150.3 ms for MASt3R.
- On UDBD, Sync-NeRF (0.2 ms median) and MASt3R (7.4 ms) beat VisualSync (5.9 ms median is second best; mean 20.2 ms vs Sync-NeRF's 0.4 ms), so VisualSync is not best on all datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use VisualSync instead of Uni4D* | Video synchronization · Egohumans | median synchronization error (ms): 222.1 -> 46.6 | Helped |
| use VisualSync instead of MAST3R | Video synchronization · CMU Panoptic | median synchronization error (ms): 58.1 -> 41.5 | Helped |
| use VisualSync instead of MAST3R | Video synchronization · 3D-POP | median synchronization error (ms): 72.2 -> 77.8 | Hurt |
| use VisualSync instead of Sync-NeRF* | Video synchronization · UDBD | median synchronization error (ms): 0.2 -> 5.9 | Hurt |
| use Sampson energy instead of cosine energy (ablation) | VisualSync · Egohumans | median synchronization error (ms): 94.6 -> 46.6 | Helped |
| use Sampson energy instead of algebraic energy (ablation) | VisualSync · Egohumans | median synchronization error (ms): 57.9 -> 46.6 | Helped |
| use Sampson energy instead of epipolar energy (ablation) | VisualSync · Egohumans | median synchronization error (ms): 35.4 -> 46.6 | Hurt |
| use IRLS instead of least squares (ablation) | VisualSync · Egohumans | median synchronization error (ms): 118.0 -> 46.6 | Helped |
| remove spurious-pair detection (ablation) | VisualSync · Egohumans | median synchronization error (ms): 46.6 -> 111.5 | Hurt |
| downsample input from 30 fps to 15 fps (ablation) | VisualSync · CMU Panoptic | median synchronization error (ms): 41.5 -> 45.6 | Mixed |
Surprising findings
- Sync-NeRF achieves the best UDBD errors, and MASt3R alone is a strong baseline despite not being trained for synchronization.
- Estimated VGGT camera poses yield 46.6 ms median error versus 28.6 ms with ground-truth inputs, yet remain usable.
Limitations
- Requires a subset of reliable camera poses from VGGT and is sensitive to upstream segmentation/correspondence errors.
- Cannot handle clips with non-uniform motion speed (e.g., alternating slow and fast motion).
- Pairwise estimation scales O(N^2) with number of videos; runtime is several hours per scene.
Experimental setup
Datasets: CMU Panoptic Egohumans 3D-POP UDBD
Benchmarks: Egohumans CMU Panoptic 3D-POP UDBD
Metrics: A@100 A@500 mean synchronization error (ms) median synchronization error (ms)
Baselines: Uni4D MASt3R Sync-NeRF
Builds on: Sync-NeRF Uni4D VGGT MASt3R CoTracker3
Closest papers by meaning
- SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints (ICLR 2025 · similarity 0.88)
- Advancing Video Synchronization with Fractional Frame Analysis: Introducing a Novel Dataset and Model (AAAI 2024 · similarity 0.85)
- Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control (NeurIPS 2024 · similarity 0.84)
- SyncLight: Single-Edit Multi-View Relighting (NeurIPS 2026 · similarity 0.83)
- Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos (AAAI 2024 · similarity 0.83)
- CoMotion: Concurrent Multi-person 3D Motion (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.