Research map / Vision-language and multimodal / Cross-modal learning
Self-Supervised MultiModal Versatile Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text modality to vision+audio training (FAC) (ablation) | MMV · UCF101 frozen HT | top-1 accuracy: 75.5 -> 84.7 | Helped |
| add audio modality to vision+text (ablation) | MMV · YC2 zero-shot R@10 | R@10: 33.6 -> 32.2 | Hurt |
| add text modality (FAC vs VA only) (ablation) | MMV S3D-G · Kinetics600 linear | top-1 accuracy: +9.2% | Helped |
| FAC merging instead of shared space (ablation) | MMV · HMDB51 frozen HT+AS | top-1 accuracy: 60.2 -> 62.5 | Helped |
| FAC merging instead of shared space (ablation) | MMV · ESC-50 frozen HT+AS | top-1 accuracy: 88.5 -> 88.0 | Hurt |
| NCE loss instead of logistic loss (ablation) | MMV vision+audio · UCF101 frozen | top-1 accuracy: 69.9 -> 71.1 | Helped |
| NCE loss instead of logistic loss (ablation) | MMV vision+audio · ESC-50 frozen | top-1 accuracy: 80.7 -> 80.0 | No clear effect |
| add non-linear projection head on vision (ablation) | MMV vision+audio · UCF101 frozen | top-1 accuracy: 71.1 -> 75.5 | Helped |
| non-linear projection head on audio/text too (ablation) | MMV · HMDB51 frozen VT | top-1 accuracy: 55.9 -> 54.4 | Hurt |
| TSM ResNet50 backbone instead of S3D-G (ablation) | MMV · UCF101 frozen VA | top-1 accuracy: 71.1 -> 75.8 | Helped |
| add audio temporal jitter (desynchronize) (ablation) | MMV vision+audio · UCF101 frozen | top-1 accuracy: 71.1 -> 67.6 | Hurt |
| add standard visual augmentation (ablation) | MMV vision+audio · UCF101 frozen | top-1 accuracy: 70.6 -> 71.1 | Helped |
Closest papers by meaning
- Parameter Efficient Multimodal Transformers for Video Representation Learning (ICLR 2021 · similarity 0.89)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.89)
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.87)
- Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion (ICML 2024 · similarity 0.87)
- SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding (CVPR 2025 · similarity 0.87)
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.