Research map / Video / Action recognition
Contrastive Learning of Global and Local Video Representations
Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the spatially-global/temporally-local objective L^l to L^g (ablation) | global-local audio-visual contrastive model · LRS2 | WER: 70.9 -> 47.6 | Improved |
| add the L^g objective to L^l (ablation) | global-local audio-visual contrastive model · LRW | top-1 accuracy: 65.1 -> 88.9 | Improved |
| use two visual encoders (slow and fast) instead of only E_v^l (ablation) | global-local model · LRW | top-1 accuracy: 81.2 -> 88.9 | Improved |
| use features from both subspaces downstream instead of z_v^l only (ablation) | global-local model · HMDB51 | accuracy: 58.3 -> 61.3 | Improved |
| use spatial attention pooling from audio-visual map instead of full-frame features (ablation) | global-local model · DFDC | AUC: 68.1 -> 95.9 | Improved |
| use lip/face crop instead of attention pooling on full frame (ablation) | global-local model · LRW | top-1 accuracy: 85.1 -> 89.2 | Improved |
| sum over multiple positive pairs with no pooling instead of average pooling single pair in L^l (ablation) | L^l objective · LRS2/LRW/DFDC | WER/accuracy/AUC: LRW 79.2 -> 83.7, DFDC 88.9 -> 90.1, LRS2 40.4 -> 47.8 | Mixed |
| audio window M=1 vs M in {3,5} (ablation) | L^l objective · downstream tasks | performance | Hurt |
| use global-local contrastive objective instead of MoCo | video pretraining · LRW | top-1 accuracy: 61.2 -> 83.7 | Improved |
Closest papers by meaning
- Audio-Visual Contrastive Learning with Temporal Self-Supervision (AAAI 2023 · similarity 0.88)
- Robust Audio-Visual Contrastive Learning for Proposal-Based Self-Supervised Sound Source Localization in Videos (TPAMI 2024 · similarity 0.88)
- Active Contrastive Learning of Audio-Visual Video Representations (ICLR 2021 · similarity 0.88)
- On Compositions of Transformations in Contrastive Self-Supervised Learning (ICCV 2021 · similarity 0.87)
- Robust Audio-Visual Instance Discrimination (CVPR 2021 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
