Research map / Label-efficient and robust learning / Self-supervised and contrastive
Learning from Synchronization: Self-Supervised Uncalibrated Multi-View Person Association in Challenging Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Self-MVA learns uncalibrated, geometry-aware multi-view person association without annotations, using a cross-view image synchronization pretext task plus re-projection and pairwise edge constraints.
Problem
Multi-view person association in challenging scenes where persons look similar, where existing methods either need identity/box annotations or calibrated camera parameters.
Approach
An encoder-decoder maps each person's box and a learnable camera embedding (no calibration) into a shared geometric feature space; it is trained by distinguishing synchronized from non-synchronized cross-view image pairs, with Hungarian matching bridging instance and image distances, plus re-projection and pseudo-edge linear constraints.
Key results
- On WILDTRACK, Self-MVA reaches IPAA-100 of 54.64 versus 35.48 for fully-supervised ViT-P3DE, with higher precision and recall.
- On MVOR, Self-MVA achieves AP 86.50, above fully-supervised ViT-P3DE (81.32) and GNN-CCA (84.70).
- On SOLDIERS, Self-MVA achieves AP 79.13 versus 56.90 for OSNet + ESC, the best unsupervised baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Self-MVA instead of ViT-P3DE | WILDTRACK association · WILDTRACK | IPAA-100: 35.48 -> 54.64 | Helped |
| use Self-MVA instead of ViT-P3DE | MVOR association · MVOR | ACC: 89.19 -> 89.38 | Helped |
| use Self-MVA instead of ViT-P3DE | MVOR association · MVOR | IPAA-100: 84.92 -> 83.92 | Hurt |
| use Self-MVA instead of OSNet + ESC | SOLDIERS association · SOLDIERS | IPAA-100: 52.12 -> 87.64 | Helped |
| add cross-view image synchronization (ablation) | Self-MVA training · WILDTRACK | AP: 4.40 -> 54.67 | Helped |
| add multi-view re-projection (ablation) | Self-MVA training · WILDTRACK | AP: 4.23 -> 54.67 | Helped |
| add pairwise edge association (ablation) | Self-MVA training · WILDTRACK | AP: 54.67 -> 56.68 | Helped |
| add Re-ID feature during training and testing (ablation) | Self-MVA · WILDTRACK | IPAA-100: 46.79 -> 54.64 | Helped |
| add Re-ID feature only during testing (ablation) | Self-MVA · WILDTRACK | AP: 56.36 -> 57.93 | Helped |
| train using ground-truth bounding boxes (ablation) | Self-MVA · WILDTRACK | IPAA-100: 54.64 -> 86.67 | Helped |
| train using ground-truth bounding boxes and identity labels (ablation) | Self-MVA · WILDTRACK | AP: 56.68 -> 89.77 | Helped |
Surprising findings
- Without re-projection or synchronization losses the model fails to converge in crowded scenes, and adding the Re-ID feature only at test time gives a smaller gain than training with it.
- AP and FPR-95 do not fully reflect association quality because they ignore the one-match-per-person constraint, so IPAA metrics are needed.
Limitations
- Not applicable to scenes with moving cameras.
- Relies on a Re-ID model and off-the-shelf detector for pseudo bounding boxes.
- On MVOR, FPR-95 is worse than ViT-P3DE and GNN-CCA, and on WILDTRACK AP is below ViT-P3DE and ASNet.
Experimental setup
Datasets: WILDTRACK MVOR SOLDIERS
Benchmarks: WILDTRACK MVOR SOLDIERS
Metrics: AP FPR-95 precision recall accuracy IPAA-X
Baselines: OSNet OSNet + ESC MvMHAT ASNet GNN-CCA ViT-P3DE
Builds on: OSNet Hungarian matching ViT-P3DE Learning from multi-view synchronization (Pose synchronization self-supervision) MvMHAT
Closest papers by meaning
- Tracklet Self-Supervised Learning for Unsupervised Person Re-Identification (AAAI 2020 · similarity 0.86)
- Simple Pose: Rethinking and Improving a Bottom-up Approach for Multi-Person Pose Estimation (AAAI 2020 · similarity 0.86)
- Divide-and-Regroup Clustering for Domain Adaptive Person Re-identification (AAAI 2022 · similarity 0.86)
- PoseTrack21: A Dataset for Person Search, Multi-Object Tracking and Multi-Person Pose Tracking (CVPR 2022 · similarity 0.85)
- REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision (NeurIPS 2021 · similarity 0.85)
- SelfPose3d: Self-Supervised Multi-Person Multi-View 3d Pose Estimation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.