Research map / Vision-language and multimodal / Cross-modal learning
Principled Multimodal Representation Learning
Xiaohao Liu, Xiaobo Xia, See Kiong Ng, Tat‐Seng Chua
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add instance-wise regularization on leading eigenvectors (ablation) | PMRL · MSR-VTT text-to-video (zero-shot) | Recall@1: 53.7 -> 54.5 | Improved |
| add instance-wise regularization on leading eigenvectors (ablation) | PMRL · DiDeMo text-to-video (zero-shot) | Recall@1: 50.2 -> 50.6 | Improved |
| add instance-wise regularization on leading eigenvectors (ablation) | PMRL · ActivityNet text-to-video (zero-shot) | Recall@1: 53.6 -> 56.0 | Improved |
| add instance-wise regularization on leading eigenvectors (ablation) | PMRL · VATEX text-to-video (zero-shot) | Recall@1: 80.0 -> 80.5 | Improved |
| add instance-wise regularization to GRAM-style volume loss (ablation) | GRAM · ActivityNet text-to-video (zero-shot) | Recall@1: 20.1 -> 54.5 | Improved |
| optimize only the top singular value instead of top-2 (sigma1+sigma2) (ablation) | PMRL · VATEX text-to-video (zero-shot) | Recall@1: 79.6 -> 80.5 | Improved |
| optimize only the top singular value instead of top-2 (sigma1+sigma2) (ablation) | PMRL · AudioCaps text-to-audio | Recall@1: 34.8 -> 36.1 | Improved |
| optimize only the top singular value instead of top-2 (sigma1+sigma2) (ablation) | PMRL · AudioCaps audio-to-text | Recall@1: 33.9 -> 34.9 | Hurt |
| replace GRAM volume loss with singular-value softmax loss | PMRL vs GRAM · ABIDE autism classification | AUC: 63.9 -> 80.5 | Improved |
| replace GRAM with PMRL on fine-tuned MSR-VTT retrieval (vs VAST) | PMRL · MSR-VTT text-to-video (fine-tuned) | Recall@1: 64.4 -> 61.2 | Hurt |
| add PMRL objective to VAST on VATEX fine-tuned retrieval | VAST · VATEX text-to-video (fine-tuned) | Recall@1: 83.1 -> 84.1 | Improved |
| add PMRL objective to VAST on Houston13 | VAST · Houston13 hyperspectral classification | overall accuracy: 7.17 -> 26.51 | Improved |
Earlier methods this paper compares with
Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.
- VAST — VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset (NeurIPS 2023) · better on 13 of 14 shared results
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.91)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.90)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.90)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.89)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
