Research map / Video / Action recognition
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Introduce hard negatives after an initial easy-negative training stage (ablation) | AVTS · Kinetics test set (easy negatives) | AVTS accuracy: 69.0 -> 78.4 | Helped |
| Use 75% easy and 25% hard negatives from the start (ablation) | AVTS · Kinetics test set (easy negatives) | AVTS accuracy: 69.0 -> 58.9 | Hurt |
| Use hard negatives alone from the start (ablation) | AVTS · Kinetics test set (easy negatives) | AVTS accuracy: 69.0 -> 52.3 | Hurt |
| Add hard negatives during a second training stage after easy-negative training (ablation) | AVTS downstream features and models · UCF101, HMDB51, ESC-50, DCASE2014 | accuracy: UCF101 77.1 -> 85.8; HMDB51 46.4 -> 56.9; ESC-50 70.6 -> 82.3; DCASE 89.2 -> 94.1 | Helped |
| Pretrain MC3 with AVTS rather than train from scratch | MC3 · UCF101 and HMDB51 | accuracy: UCF101 69.1 -> 85.8; HMDB51 43.9 -> 56.9 | Helped |
| Increase AVTS pretraining data from Kinetics to full AudioSet | MC3 · UCF101 | accuracy: 85.8 -> 89.0 | Helped |
| Increase AudioSet AVTS pretraining examples (ablation) | AVTS audio features · ESC-50 | accuracy: 77.4 -> 78.9 -> 79.5 -> 80.6 | Helped |
| Replace cross-entropy optimization from scratch with contrastive loss (ablation) | AVTS · AVTS training | optimization convergence | Helped |
| Fine-tune the contrastive-loss model using cross-entropy for downstream feature learning (ablation) | AVTS · Action recognition and audio classification | downstream performance | No clear effect |
| Use super-hard negatives during training (ablation) | AVTS · AVTS | AVTS performance | Hurt |
| Use thresholded contrastive distance instead of a cross-entropy-finetuned classifier (ablation) | AVTS · Kinetics test set with mixed easy and hard negatives | AVTS accuracy: 65.6% vs 70.3% | Hurt |
| Use both audio and video streams rather than video only (ablation) | AVTS action recognition model · UCF101 | accuracy: 85.8% -> 87.0% | Helped |
Closest papers by meaning
- Audio-Visual Contrastive Learning with Temporal Self-Supervision (AAAI 2023 · similarity 0.91)
- Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity (AAAI 2023 · similarity 0.91)
- Enhancing Audio-Visual Association with Self-Supervised Curriculum Learning (AAAI 2021 · similarity 0.90)
- Robust Audio-Visual Instance Discrimination (CVPR 2021 · similarity 0.89)
- TIM: A Time Interval Machine for Audio-Visual Action Recognition (CVPR 2024 · similarity 0.87)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.