Research map / 3D vision / Human pose estimation
Advancing Video Synchronization with Fractional Frame Analysis: Introducing a Novel Dataset and Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the IFID dataset with fractional-frame (inter- and intra-frame) time-offset labels for unsynchronized multi-view videos and the InSynFormer Transformer model, trained with hierarchical cross-entropy loss, for fractional-frame video synchronization.
Problem
Video synchronization across multiple unsynchronized cameras typically only achieves frame-level accuracy, because labeled offsets are integer multiples of the frame duration; fractional (intra-frame) offsets are ignored, which hurts multi-view 3D pose estimation on naturally recorded videos.
Approach
Label time offsets as a hierarchy of inter-frame and intra-frame (0.1-frame resolution) intervals using a timer-based annotation protocol, and synchronize with a Transformer that uses temporal self-attention within views, cross-view attention between views on per-joint pose heatmaps, and a hierarchical cross-entropy loss.
Key results
- On IFID, InSynFormer reaches 80.86% Accex@1 and 0.83 frame error, versus 79.93% and 0.87 for SeSyn-Net (inter-frame only comparison).
- Intra-frame accuracy on IFID reaches 61.30% Accin@1 and 90.69% Accin@3 with hierarchical cross-entropy, compared with 31.78% and 70.12% for SynNet with the same loss.
- Ablation shows TSA+CVA raises Accex@3 from 91.22% (MHSA) to 94.35%, and HXE loss improves Accin@1 from 49.84% to 61.30% over CE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use InSynFormer instead of SynNet | video synchronization · IFID | Accex@1: 60.41% -> 80.86% | Helped |
| use InSynFormer instead of SeSyn-Net | video synchronization · IFID | Accex@1: 79.93% -> 80.86% | Helped |
| use InSynFormer instead of CNNSiamese | video synchronization · IFID | Frm.err.: 2.04 -> 0.83 | Helped |
| add hierarchical cross-entropy loss to SynNet (ablation) | SynNet on IFID · IFID | Accin@1: 20.56% -> 31.78% | Helped |
| add hierarchical cross-entropy loss to InSynFormer with MHSA (ablation) | InSynFormer with MHSA on IFID · IFID | Accin@1: 19.78% -> 35.65% | Helped |
| add hierarchical cross-entropy loss to InSynFormer with TSA and CVA (ablation) | InSynFormer on IFID · IFID | Accin@1: 49.84% -> 61.30% | Helped |
| replace MHSA with TSA and CVA (ablation) | InSynFormer with cross-entropy loss · IFID | Accin@1: 19.78% -> 49.84% | Helped |
| replace MHSA with TSA and CVA (ablation) | InSynFormer with hierarchical cross-entropy loss · IFID | Accin@1: 35.65% -> 61.30% | Helped |
| replace MHSA with TSA and CVA (ablation) | InSynFormer · IFID | Frm.err.: 1.13 -> 0.83 | Helped |
| use hierarchical cross-entropy loss instead of cross-entropy loss (ablation) | InSynFormer with TSA and CVA · IFID | Frm.err.: 0.83 -> 0.83 | No clear effect |
Surprising findings
- Plain cross-entropy loss causes intra-frame accuracy to decline when combined with the inter-frame objective, while hierarchical cross-entropy keeps both levels well trained.
Limitations
- Still a slight residual error from classifying fractional intervals rather than regressing them directly.
- Relies on human pose in the video; scenes without people cannot be synchronized.
- Existing multi-view 3D pose estimators cannot process desynchronized videos, so intra-frame offsets are not exploited downstream.
Experimental setup
Datasets: IFID NTU-SYN CMU-SYN NTU RGB+D CMU Panoptic Studio
Benchmarks: IFID NTU-SYN CMU-SYN
Metrics: Accex@1 Accex@3 Accin@1 Accin@3 Accin@5 Frame error (Frm.err.)
Baselines: SynNet SeSyn-Net CNNSiamese
Builds on: Transformer hierarchical cross-entropy loss SynNet SeSyn-Net
Closest papers by meaning
- Can Generative Video Models Help Pose Estimation? (CVPR 2025 · similarity 0.86)
- Visual Sync: Multi‑Camera Synchronization via Cross‑View Object Motion (NeurIPS 2025 · similarity 0.85)
- Multi-View 3D Human Pose Estimation with Weakly Synchronized Images (AAAI 2025 · similarity 0.85)
- PoseCrafter: Extreme Pose Estimation with Hybrid Video Synthesis (NeurIPS 2025 · similarity 0.84)
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.84)
- Dynamic Camera Poses and Where to Find Them (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.