Research map / Video / Video segmentation and tracking
No More Shortcuts: Realizing the Potential of Temporal Self-Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes frame-level temporal self-supervision (out-of-order frame localization and time-varying skiprate prediction) with frame-wise augmentation to remove local-appearance shortcuts, training a transformer over frozen-initialized image encoder features pre-trained on Kinetics400.
Problem
Existing temporal self-supervised video pretext tasks are too simple (near-saturated training accuracy) and can be solved via local appearance shortcuts, so they fail to learn high-level temporal/object dynamics features.
Approach
Reformulate temporal pretext tasks as per-frame classification (out-of-order frame localization and time-varying skiprate prediction) and apply independent per-frame spatial augmentations to break local-patch shortcuts, on top of a contrastively pre-trained ViT frame encoder extended with a temporal transformer.
Key results
- ViT-B achieves 85.1 R@1 on UCF101 and 49.4 R@1 on HMDB51 video retrieval, improving Top-1 over prior SOTA by 3.3% and 8.2%.
- ViT-L reaches 91.5 linear top-1 on U101 and 62.5 on H51, and 53.1 on NTU60 linear probing.
- Frame-wise augmentation removal drops UCF101 linear accuracy from 89.90 to 88.05-ish range (2-4% drop) in ablations on UCF101 pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use frame-level rather than clip-level OFL (ablation) | OFL temporal self-supervision · UCF101 | R@1: 80.85 -> 83.90 | Helped |
| use frame-level rather than clip-level TSP (ablation) | TSP temporal self-supervision · HMDB51 | R@1: 44.85 -> 46.14 | Helped |
| add framewise augmentation (ablation) | full temporal SSL objectives · UCF101 | R@1: 82.50 -> 84.68 | Helped |
| add OFL to video contrastive learning (ablation) | video representation learning · UCF101 | R@1: 80.15 -> 83.90 | Helped |
| add TSP to video contrastive learning (ablation) | video representation learning · UCF101 | R@1: 80.15 -> 82.73 | Helped |
| randomize outlier rate over U([0.0, 0.50]) rather than fixing it at 0.25 (ablation) | OFL · UCF101 | R@1: 83.40 -> 84.01 | Helped |
| restrict replacement-frame distance to a maximum of 64 (ablation) | OFL · UCF101 | R@1: 84.01 -> 84.68 | Helped |
| use 3-way skiprates {1,4,8} rather than 2-way {1,4} (ablation) | TSP · UCF101 | R@1: 83.62 -> 84.01 | Helped |
| add framewise color jittering to cropping (ablation) | framewise augmentation · UCF101 | R@1: 84.68 -> 84.24 | Hurt |
| add framewise horizontal flipping to cropping (ablation) | framewise augmentation · HMDB51 | R@1: 50.20 -> 48.08 | Hurt |
| add temporal features to static appearance features (ablation) | video representation · HMDB51 | Lin: 55.90 -> 62.50 | Helped |
| increase temporal encoder layers from 1 to 6 (ablation) | Video Transformer Network · HMDB51 | R@1: 47.10 -> 50.20 | Helped |
Surprising findings
- A local 32x32 patch model achieves 78.2 OFL mAP without frame-wise augmentation, indicating shortcuts based on local appearance.
- Horizontal flipping as a frame-wise augmentation reduces performance by 1-2%.
Limitations
- Evaluated mainly with fixed features rather than full fine-tuning
- Relies on ImageNet-pretrained image encoder initialization
Experimental setup
Datasets: Kinetics400 ImageNet-1k UCF101
Benchmarks: UCF101 HMDB51 Kinetics400 SSv2 NTU60 Charades HVU DAVIS-2017 JHMDB CASIA-B
Metrics: top-1 accuracy R@1 R@5 mAP J&F PCK rank-1 accuracy
Baselines: MoCo VCOP Pace Pred SeLaVi CoCLR CVRL rhoBYOL SVT VideoMAE MaCLR MCL CtP
Builds on: Video Transformer Network MUGS contrastive learning VideoMoCo
Closest papers by meaning
- Time Is MattEr: Temporal Self-supervision for Video Transformers (ICML 2022 · similarity 0.89)
- Hierarchical Self-Supervised Representation Learning for Movie Understanding (CVPR 2022 · similarity 0.89)
- Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics (CVPR 2019 · similarity 0.89)
- Learning by Aligning Videos in Time (CVPR 2021 · similarity 0.89)
- When the Future Becomes the Past: Taming Temporal Correspondence for Self-supervised Video Representation Learning (CVPR 2025 · similarity 0.88)
- Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.