Research map / Vision-language and multimodal / Cross-modal learning
Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add asynchronous cross-modal loss to the intra-modal and synchronous-loss objective (ablation) | CrissCross pretrained on Kinetics400 · UCF101 | top-1 accuracy: 75.8 -> 79.9 | Helped |
| Add asynchronous cross-modal loss to the intra-modal and synchronous-loss objective (ablation) | CrissCross pretrained on Kinetics400 · ESC50 | top-1 accuracy: 78.5 -> 82.0 | Helped |
| Add asynchronous cross-modal loss to the intra-modal and synchronous-loss objective (ablation) | CrissCross pretrained on Kinetics-Sound · UCF101 | top-1 accuracy: 73.8 -> 74.8 | Helped |
| Add asynchronous cross-modal loss to the intra-modal and synchronous-loss objective (ablation) | CrissCross pretrained on Kinetics-Sound · ESC50 | top-1 accuracy: 78.0 -> 79.0 | Helped |
| Use all intra-modal, synchronous, and asynchronous losses rather than synchronous loss alone (ablation) | CrissCross · UCF101, Kinetics-Sound pretraining | top-1 accuracy: 70.1 -> 74.8 | Helped |
| Use all intra-modal, synchronous, and asynchronous losses rather than synchronous loss alone (ablation) | CrissCross · ESC50, Kinetics-Sound pretraining | top-1 accuracy: 75.8 -> 79.0 | Helped |
| Relax temporal alignment mildly, with the two views sharing 50% overlap (ablation) | CrissCross · UCF101 and ESC50 | top-1 accuracy | Helped |
| Use medium temporal relaxation with adjacent frame sequences and audio segments (ablation) | CrissCross · UCF101 and ESC50 | top-1 accuracy | Hurt |
| Set predictor learning rate equal to the backbone learning rate instead of using a higher predictor rate (ablation) | CrissCross · UCF101 | top-1 accuracy: 74.8 -> 59.0 | Hurt |
| Use a common predictor instead of separate audio and visual predictors (ablation) | CrissCross · UCF101 | top-1 accuracy: 74.8 -> 73.6 | Hurt |
| Use a two-layer projector instead of the default three-layer projector (ablation) | CrissCross · UCF101 | top-1 accuracy: 72.4 -> 74.8 | Hurt |
| Use a two-layer projector instead of the default three-layer projector (ablation) | CrissCross · ESC50 | top-1 accuracy: 75.0 -> 79.0 | Hurt |
Closest papers by meaning
- Audio-Visual Instance Discrimination with Cross-Modal Agreement (CVPR 2021 · similarity 0.91)
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (NeurIPS 2018 · similarity 0.91)
- Robust Audio-Visual Instance Discrimination (CVPR 2021 · similarity 0.90)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.87)
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.87)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.