Research map / Video / Video segmentation and tracking
When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-CoRe, a self-supervised video representation framework that samples past and future auxiliary frames (sandwich sampling) and restores masked patches in latent space via a Patch Matching Module within a DINO-style self-distillation setup.
Problem
Masked video modeling with random temporal sampling yields uncertain reconstructions, and pixel-space reconstruction retains excessive low-level detail, hurting downstream video representation quality.
Approach
Sample a past and a future frame around the current frame so temporal correspondence squeezes reconstruction uncertainty from both sides, and restore masked patches in latent space by cross/self-attention patch matching against auxiliary-frame embeddings, supervised by a self-distillation teacher with an added past-future alignment loss.
Key results
- On Kinetics-400 pre-training with ViT-S/16 (400 epochs), T-CoRe reaches 64.7 J&Fm on DAVIS-2017 versus 63.5 for iBOT and 63.0 for DINO v2.
- With ViT-B/16 backbone (200 epochs), T-CoRe achieves 66.4 J&Fm on DAVIS-2017 and 64.6 mIoU on VIP, outperforming DINO v2 (64.9 and 63.3).
- Ablation shows the auxiliary branch with PMM and past-future alignment loss Lpf adds about 2.6% and 1.0% J&Fm gains respectively on DAVIS-2017.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a past frame to the auxiliary branch (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 61.9 -> 63.4 | Helped |
| add a future frame to the auxiliary branch (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 61.9 -> 63.0 | Helped |
| add Lpf to align representations from past and future frames (ablation) | T-CoRe with past and future auxiliary frames · DAVIS-2017 | J &Fm: 63.5 -> 64.4 | Helped |
| add the auxiliary branch to the self-distillation architecture with fixed spatial and temporal alignment (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 61.9 -> 64.4 | Helped |
| use past and future offsets [0.20, 0.30] instead of [0.15, 0.25] (ablation) | T-CoRe sandwich sampling · DAVIS-2017 | J &Fm: 64.4 -> 62.8 | Hurt |
| sample both auxiliary frames from the past instead of one from each side (ablation) | T-CoRe sandwich sampling · DAVIS-2017 | J &Fm: 64.4 -> 63.4 | Hurt |
| sample both auxiliary frames from the future instead of one from each side (ablation) | T-CoRe sandwich sampling · DAVIS-2017 | J &Fm: 64.4 -> 63.3 | Hurt |
| set mask probability to 0.1 instead of 0.5 (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 64.4 -> 60.1 | Hurt |
| set mask probability to 0.7 instead of 0.5 (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 64.4 -> 62.7 | Hurt |
| use mask ratios 0.3–0.7 instead of 0.1–0.5 (ablation) | T-CoRe · DAVIS-2017 | J &Fm: 64.4 -> 63.7 | Hurt |
| pre-train T-CoRe instead of DINO v2 | ViT-S/16 on DAVIS-2017 · DAVIS-2017 | J &Fm: 63.2 -> 64.1 | Helped |
| pre-train T-CoRe instead of iBOT | ViT-S/16 on Kinetics-400, evaluated on DAVIS-2017 · DAVIS-2017 | J &Fm: 63.5 -> 64.7 | Helped |
Surprising findings
- Using only a future auxiliary frame yields slightly lower results than a past frame, attributed to the anisotropy of time.
Limitations
- Gains over DINO v2 and iBOT on DAVIS-2017 are modest (about 1.2-1.7 J&Fm points).
- Hyperparameters tuned only for lambda1 and lambda2; sampling offsets and masking scheme require ablation-driven selection.
- Evaluated only on frame-level dense propagation tasks with semi-supervised protocol; no evaluation on action recognition or other video-level tasks reported.
Experimental setup
Datasets: Kinetics-400 ImageNet-1k DAVIS-2017 VIP JHMDB
Benchmarks: DAVIS-2017 VIP JHMDB
Metrics: J&Fm Jm Fm mIoU PCK@0.1 PCK@0.2
Baselines: DINO iBOT DINO v2 SiamMAE CropMAE RSP MAE-ST DropMAE VideoMAE CrOC
Builds on: DINO iBOT DINO v2 SiamMAE MAE
Closest papers by meaning
- Time Is MattEr: Temporal Self-supervision for Video Transformers (ICML 2022 · similarity 0.88)
- Spatial-Then-Temporal Self-Supervised Learning for Video Correspondence (CVPR 2023 · similarity 0.88)
- Joint-task Self-supervised Learning for Temporal Correspondence (NeurIPS 2019 · similarity 0.88)
- No More Shortcuts: Realizing the Potential of Temporal Self-Supervision (AAAI 2024 · similarity 0.88)
- Learning Correspondence From the Cycle-Consistency of Time (CVPR 2019 · similarity 0.88)
- Learning by Aligning Videos in Time (CVPR 2021 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.