Research map / Vision-language and multimodal / Cross-modal learning
Open-set Cross Modal Generalization via Multimodal Unified Representation
Hai Huang, Yan Xia, Shulei Wang, Hanting Wang, Minghui Fang, Shengpeng Ji, Sashuai Zhou, Tao Jin, Zhou Zhao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DCID with MICU on open-set cross-modal generalization | OSCMG · AVE, UCF, UCF(v)<->VGG(a) | HOS: AVE Split1 V->A 51.54 -> 54.39 | Improved |
| use MICU on VGG(a)->UCF(v) Split2 | OSCMG · UCF(v)<->VGG(a) Split2 A->V | HOS: 62.49 (DCID) -> 61.17 | Hurt |
| add L_fine to L_coarse (ablation) | MICU · OSCMG AVE Split1 V->A | HOS: 49.57 -> 54.05 | Improved |
| add L_cujp to L_coarse (ablation) | MICU · OSCMG AVE Split1 V->A | HOS: 49.57 -> 54.09 | Improved |
| use L_fine or L_cujp without L_coarse (ablation) | MICU · OSCMG AVE Split1 V->A | HOS: 9.17 (L_fine only), 11.46 (L_cujp only) vs 49.57 (L_coarse only) | Hurt |
| use CUJP instead of MMJP jigsaw puzzles (ablation) | MICU · OSCMG and CMG averages | OSCMG avg: 52.00 (MMJP6) -> 52.56 (CUJP4), 54.29 (CUJP8) | Improved |
| use MMJP jigsaw instead of no jigsaw (ablation) | MICU · OSCMG and CMG classification | performance | Hurt |
| increase CUJP splits from 4 to 8 (ablation) | CUJP · OSCMG avg | OSCMG average: 52.56 -> 54.29; memory 9.07GB -> 13.30GB | Improved |
| use stronger backbones (Swin-V2-L, HTS-AT) (ablation) | OSCMG methods · AVE Split1 V->A | HOS: MICU 54.39 -> 57.76 | Improved |
| apply the same mask positions across paired modalities rather than different positions (ablation) | FCMI · OSCMG/CMG | performance | Improved |
| use codebook size 400 instead of 256/512/800/1024 (ablation) | MICU · OSCMG/CMG | performance | Improved |
Closest papers by meaning
- Cross-modal Representation Flattening for Multi-modal Domain Generalization (NeurIPS 2024 · similarity 0.89)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment (CVPR 2025 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.86)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.86)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
