Research map / Vision-language and multimodal / Cross-modal learning
Enhancing Audio-Visual Association with Self-Supervised Curriculum Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stage-II role exchange and cross-modal transfer after stage-I (ablation) | SSCL · UCF-101 | accuracy: 81.4 -> 82.6 | Helped |
| Add stage-II role exchange and cross-modal transfer after stage-I (ablation) | SSCL · HMDB-51 | accuracy: 47.7 -> 49.9 | Helped |
| Add stage-II role exchange and cross-modal transfer after stage-I (ablation) | SSCL · ESC-50 | accuracy: 85.8 -> 88.3 | Helped |
| Add stage-II role exchange and cross-modal transfer after stage-I (ablation) | SSCL · DCASE | accuracy: 91.0 -> 93.0 | Helped |
| Use larger video input resolution, from 16x112x112 to 16x224x224 (ablation) | SSCL-stage-II · UCF-101 | accuracy: 82.6 -> 84.3 | Helped |
| Use larger video input resolution, from 16x112x112 to 16x224x224 (ablation) | SSCL-stage-II · HMDB-51 | accuracy: 49.9 -> 54.1 | Helped |
| Increase temporal input from 16 to 32 frames at 224x224 resolution (ablation) | SSCL-stage-II · UCF-101 | accuracy: 84.3 -> 87.1 | Helped |
| Increase temporal input from 16 to 32 frames at 224x224 resolution (ablation) | SSCL-stage-II · HMDB-51 | accuracy: 54.1 -> 57.6 | Helped |
| Pre-train by directly optimizing audio-visual transfer without curriculum calibration (ablation) | SSCL · UCF-101 | accuracy | Hurt |
| Use the two-stage curriculum, including within-modality calibration and subsequent cross-modal transfer (ablation) | SSCL · UCF-101 and ESC-50 | accuracy | Helped |
| Replace the contrastive pre-training task with 3DRotNet or Clip-order (ablation) | SSCL video pre-training · UCF-101 | accuracy | Hurt |
Closest papers by meaning
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization (NeurIPS 2018 · similarity 0.90)
- Audio-Visual Contrastive Learning with Temporal Self-Supervision (AAAI 2023 · similarity 0.89)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
- Few-Shot Audio-Visual Class-Incremental Learning with Temporal Prompting and Regularization (AAAI 2025 · similarity 0.86)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.85)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.