Research map / Vision-language and multimodal / Cross-modal learning
Preserving Modality Structure Improves Multi-Modal Learning
Sirnam Swetha, Mamshad Nayeem Rizve, Nina Shvetsova, Hilde Kuehne, Mubarak Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SSPC loss with reconstruction loss plus contrastive loss (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 25.1 -> 23.1 | Hurt |
| add cross-modal SSPC loss (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 22.7 -> 25.1 | Improved |
| add cross-modal SSPC loss (ablation) | Semantic-structure-preserving consistency method · YouCook2 zero-shot retrieval | R@5: 36.9 -> 39.4 | Improved |
| add anchor consistency between modality-specific and joint embedding spaces (SSPC loss) (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 23.8 -> 25.1 | Improved |
| use proposed Multi-SK algorithm instead of modified vanilla Sinkhorn-Knopp (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 23.4 -> 25.1 | Improved |
| increase number of anchors K from 16 to 64 (with K' fixed proportionally) (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 23.1 -> 25.1 | Improved |
| select 48 of 64 anchors instead of 32 (ablation) | Semantic-structure-preserving consistency method · MSR-VTT zero-shot retrieval | R@5: 25.1 -> 23.7 | Hurt |
| use learned multi-anchor structure preservation versus EAO baseline on out-of-domain MSR-VTT (fine-tune) | Semantic-structure-preserving consistency method · MSR-VTT t->v fine-tune | R@5: 47.7 -> 48.7 | Improved |
| proposed method versus EAO on zero-shot text-to-full-video retrieval | Semantic-structure-preserving consistency method · YouCook2 zero-shot full video | R@1: 62.9 -> 65.1 | Improved |
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.87)
- Cross-modal Representation Flattening for Multi-modal Domain Generalization (NeurIPS 2024 · similarity 0.87)
- Generative Modeling of Class Probability for Multi-Modal Representation Learning (CVPR 2025 · similarity 0.87)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.87)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.87)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
