Research map / Video / Action recognition
Multimodal Clustering Networks for Self-Supervised Learning From Unlabeled Videos
Brian Chen, Andrew Rouditchenko, Kevin Duarte, Hilde Kuehne, Samuel Thomas, Angie Boggust, Rameswar Panda, Brian Kingsbury, Rogerio Feris, David Harwath, James Glass, Michael Picheny and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the clustering loss to MMS contrastive loss (ablation) | MCN · YouCook2 R@10 | R@10: 43.7 -> 44.3 | Improved |
| add the clustering loss to MMS contrastive loss (ablation) | MCN · MSR-VTT R@10 | R@10: 32.9 -> 33.7 | Improved |
| add the clustering loss to MMS contrastive loss (ablation) | MCN · MiningYouTube | IoU: 22.1 -> 22.6 | Improved |
| add the reconstruction loss (ablation) | MCN · YouCook2 R@10 | R@10: 44.3 -> 45.2 | Improved |
| add the reconstruction loss (ablation) | MCN · MiningYouTube | IoU: 22.6 -> 23.1 | Improved |
| use MMS instead of NCE as contrastive loss (ablation) | MCN · YouCook2 R@10 and MSR-VTT R@10 | R@10: 39.2 -> 43.7 (YR10); 33.5 -> 32.9 (MR10) | Mixed |
| use MIL-NCE instead of NCE as contrastive loss (ablation) | MCN · YouCook2 R@10 | R@10: 39.2 -> 40.0 | No clear effect |
| use centroid target with K-means instead of joint prediction target (ablation) | MCN · MiningYouTube | IoU: 21.4 -> 23.1 | Improved |
| use joint prediction instead of swap prediction with K-means (ablation) | MCN · YouCook2 R@10 | R@10: 41.3 -> 44.3 | Improved |
| use soft instead of hard labels with Sinkhorn swap (ablation) | MCN · YouCook2 R@10 | R@10: 39.0 -> 41.8 | Improved |
| add audio modality (retrieve T to VA instead of T to V) (ablation) | MCN · YouCook2 R@1 | R@1: 8.6 -> 18.1 | Improved |
| use caption averaging instead of majority vote over clips (ablation) | MCN · YouCook2 text-to-full video retrieval | R@1: 38.8 -> 53.4 | Improved |
Closest papers by meaning
- Self-Supervised MultiModal Versatile Networks (NeurIPS 2020 · similarity 0.91)
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering (NeurIPS 2020 · similarity 0.88)
- MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation (AAAI 2025 · similarity 0.86)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.86)
- Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.86)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
