Research map / Video / Action recognition
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cross-modal exclusive pseudo-labels (XDC) instead of single-modality clustering (SDC) (ablation) | deep clustering · UCF101 split-1 | top-1 accuracy: 61.8 -> 74.2 | Helped |
| use cross-modal pseudo-labels instead of single-modality clustering (ablation) | deep clustering · ESC50 split-1 | top-1 accuracy: 66.5 -> 78.0 | Helped |
| exclusive cross-modal supervision (XDC) vs concatenated clustering (CDC) (ablation) | multi-modal deep clustering · HMDB51 split-1 | top-1 accuracy: 37.5 -> 39.0 | Helped |
| use other modality rather than a second same-modality encoder for pseudo-labels (ablation) | XDC · UCF101 split-1 | top-1 accuracy: 61.3 -> 74.2 | Helped |
| increase pretraining data from Kinetics 240K to IG-Kinetics 65M (ablation) | XDC · HMDB51 split-1 | top-1 accuracy: 39.0 -> 63.1 | Helped |
| curated (IG-Kinetics) instead of uncurated (IG-Random) pretraining data at 65M (ablation) | XDC · ESC50 split-1 | top-1 accuracy: 86.3 -> 84.8 | Hurt |
| curated instead of uncurated pretraining data at 65M (ablation) | XDC · HMDB51 split-1 | top-1 accuracy: 61.2 -> 63.1 | Helped |
| vary number of k-means clusters 64 to 1024 (ablation) | XDC · UCF101 split-1 (Kinetics pretrain) | top-1 accuracy: 73.8 / 74.2 / 72.6 | Mixed |
| replace Kinetics supervised pretraining with XDC IG-Kinetics self-supervision | R(2+1)D-18 · HMDB51 | top-1 accuracy: 65.1 -> 68.9 | Helped |
| replace supervised Kinetics features with XDC features in G-TAD | G-TAD · THUMOS14 | mAP@0.5: 36.6 -> 37.2 | Helped |
Closest papers by meaning
- XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning (AAAI 2024 · similarity 0.91)
- Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity (AAAI 2023 · similarity 0.90)
- Audio-Visual Instance Discrimination with Cross-Modal Agreement (CVPR 2021 · similarity 0.89)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.86)
- SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos (CVPR 2024 · similarity 0.85)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.