Research map / Vision-language and multimodal / Cross-modal learning
i-Code: An Integrative and Composable Multimodal Learning Framework
Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, DongDong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, Liyang Lu, Yujia Xie and 8 more
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain on dual-modality datasets instead of no pretraining (ablation) | i-Code Merge · CMU-MOSEI Sentiment | MAE: 0.529 -> 0.502 | Improved |
| Pretrain on dual-modality datasets instead of no pretraining (ablation) | i-Code Merge · CMU-MOSEI Emotion Recognition | Accuracy: 49.2 -> 49.4 | Improved |
| Use video pretraining rather than dual-dataset pretraining (ablation) | i-Code Merge · UR-FUNNY | Accuracy: 76.94 -> 79.17 | Improved |
| Use dual-dataset pretraining rather than video pretraining (ablation) | i-Code Merge · CMU-MOSEI Sentiment | MAE: 0.519 -> 0.502 | Improved |
| Combine dual-dataset and video pretraining (ablation) | i-Code Merge · CMU-MOSEI Emotion Recognition | Accuracy: 49.4 -> 49.8 | Improved |
| Replace merge-attention with co-attention (ablation) | i-Code with dual-data pretraining · CMU-MOSEI Sentiment | MAE: 0.502 -> 0.525 | Hurt |
| Replace merge-attention with co-attention (ablation) | i-Code with dual-data pretraining · CMU-MOSEI Emotion Recognition | Accuracy: 49.4 -> 50.2 | Improved |
| Add language to speech-only input (ablation) | i-Code · CMU-MOSEI Emotion Recognition | Accuracy: 47.3 -> 49.2 | Improved |
| Add vision, language, and speech rather than language and speech only (ablation) | i-Code · CMU-MOSEI Emotion Recognition | Accuracy: 49.2 -> 49.4 | Improved |
| Replace WavLM-large with HuBERT-large (ablation) | i-Code Dual+Merge · VIOLIN | Accuracy: 72.9 -> 72.1 | Hurt |
| Use learnable rather than fixed loss-combination weights (ablation) | i-Code pretraining · downstream tasks | performance | No clear effect |
| Use the pretrained multimodal fusion framework rather than no fusion pretraining (ablation) | i-Code · How2QA public test | Accuracy: 74.41 -> 75.41 | Improved |
Later papers that compared with this method
1 later accepted paper reports i-Code as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints (AAAI 2025 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception (NeurIPS 2023 · similarity 0.89)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.89)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
- Beyond Language Modeling: An Exploration of Multimodal Pretraining (ICML 2026 · similarity 0.88)
- MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning (ICML 2026 · similarity 0.88)
- Understanding Multimodal Learning From Modality Fusion and Alignment Perspectives (TPAMI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
