Research map / Video / Video segmentation and tracking
Self-Supervised Cross-View Correspondence with Predictive Cycle Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Predictive Cycle Consistency, which bootstraps object correspondence pseudolabels from SAM masks using grayscale colorization and cycle consistency, then iteratively trains ViT-B correspondence models for self-supervised correspondence across extreme view and temporal changes.
Problem
Self-supervised object correspondence across extreme viewpoint changes (EgoExo4D) and large temporal gaps in video, where existing methods assume small transformations or view overlap.
Approach
Extract object-level correspondences by running a conditional grayscale colorization model in both directions with object color augmentations and keeping only cycle-consistent object pairs as pseudolabels; then iteratively retrain ViT correspondence models on these pseudolabels, replacing colorization with the trained model for further rounds.
Key results
- On EgoExo4D test, PCC Iter 3 reaches 41.45 IoU in Exo Query versus 34.78 for DINOv2+Registers+SAM, and 29.89 IoU in Ego Query.
- On DAVIS-17 val with a 20-frame gap, PCC Iter 3 achieves 70.2 J&Fm versus 66.4 for DINO ViT-B/8.
- On EgoExo4D, adding PCC Iter 3 pseudolabels to supervised training raises Exo Query IoU from 43.70 to 47.06.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PCC Iteration 3 pseudolabels to supervised training | Ours Supervised on EgoExo4D · EgoExo4D test, Exo Query | IoU: 43.70 -> 47.06 | Helped |
| add PCC Iteration 3 pseudolabels to supervised training | Ours Supervised on EgoExo4D · EgoExo4D test, Ego Query | IoU: 38.41 -> 39.01 | Helped |
| iterate pseudolabeling and training from Iteration 1 to Iteration 2 (ablation) | PCC on DAVIS-2017 · DAVIS-2017 val, 20-frame gap | J &Fm: 64.4 -> 69.7 | Helped |
| iterate pseudolabeling and training from Iteration 2 to Iteration 3 (ablation) | PCC on DAVIS-2017 · DAVIS-2017 val, 20-frame gap | J &Fm: 69.7 -> 70.2 | Helped |
| use grayscale pretraining instead of MAE pretraining (ablation) | supervised correspondence model on EgoExo4D · EgoExo4D validation | IoU (Ego/Exo Query): 37.7/43.7 -> 31.0/34.3 | Hurt |
| use a 50M-parameter decoder instead of a 100M-parameter decoder (ablation) | supervised correspondence model on EgoExo4D · EgoExo4D validation | IoU (Ego/Exo Query): 36.9/40.2 -> 37.7/43.7 | Hurt |
| increase the temporal gap used for pseudolabel extraction (ablation) | PCC trained on Kinetics-400 · DAVIS-2017 validation | J &Fm at 30-frame gap: 65.4 -> 66.3 -> 67.7 | Helped |
Surprising findings
- Grayscale colorization pretraining on EgoExo4D degrades supervised correspondence performance relative to MAE pretraining, suggesting gains come from the full cycle-consistent pipeline rather than representation learning.
- Adding SAM at inference improves IoU and contour accuracy for self-supervised baselines but hurts location score.
Limitations
- Relies on SAM segmentations for object enumeration, and SAM selection can cut off objects on DAVIS/LVOS.
- Fine-grained mask quality is sometimes poor, according to qualitative results.
- Performance saturates after the second or third pseudolabeling iteration.
Experimental setup
Datasets: EgoExo4D correspondence benchmark Kinetics-400 DAVIS-2017 LVOS
Benchmarks: EgoExo4D correspondence benchmark DAVIS-2017 LVOS
Metrics: IoU Location Score Contour Accuracy Balanced Accuracy J&Fm Jm Fm
Baselines: SiamMAE CrocoV2 DINO DINOv2+Registers XSegTx XView-XMem Probabilistic Warp Consistency Grayscale Coloration + SAM
Builds on: SiamMAE CroCo v2 Grayscale colorization (Vondrick et al.) Segment Anything (SAM) XView-XMem
Closest papers by meaning
- Learning Correspondence From the Cycle-Consistency of Time (CVPR 2019 · similarity 0.91)
- Learning Dense Correspondence via 3D-Guided Cycle Consistency (CVPR 2016 · similarity 0.90)
- Contrastive Transformation for Self-supervised Correspondence Learning (AAAI 2021 · similarity 0.89)
- Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction (CVPR 2026 · similarity 0.89)
- Self-Supervised Spatial Correspondence Across Modalities (CVPR 2025 · similarity 0.86)
- Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.