Research map / Vision-language and multimodal / Cross-modal learning
$\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes I²MoE, an end-to-end mixture-of-experts fusion framework with four PID-inspired interaction experts (two uniqueness, synergy, redundancy) trained with perturbation-based weak interaction losses and a sample-adaptive reweighting MLP, giving local and global interpretability.
Problem
Existing multimodal fusion methods do not explicitly model heterogeneous modality interactions (uniqueness, redundancy, synergy) and output a single prediction without interpretable insight into those interactions.
Approach
Interaction experts are trained with weakly supervised losses computed from randomly masked-modality forward passes, using PID-style relations between full-input and unimodal predictions; a learned reweighting MLP combines expert outputs, and its weights provide sample-level and dataset-level interpretation.
Key results
- I²MoE-MulT reaches 65.08% accuracy on ADNI vs 59.57% for vanilla MulT (+5.5 points), and 71.91% on MOSI vs 68.80%.
- I²MoE integrated with SwitchGate improves ADNI accuracy by 5.23 points (62.28 to 67.51 reported in the transfer table).
- In a 15-participant human study, 70.4% of expert-weight ratings were positive (mostly or completely makes sense).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add I²MoE interaction experts and adaptive weighting to MulT | MulT · ADNI | Accuracy: 59.57 -> 65.08 | Helped |
| add I²MoE interaction experts and adaptive weighting to MulT | MulT · MIMIC | Accuracy: 72.42 -> 69.78 | Hurt |
| add I²MoE interaction experts and adaptive weighting to MulT | MulT · MIMIC | AUROC: 68.79 -> 68.81 | Helped |
| add I²MoE interaction experts and adaptive weighting to MulT | MulT · IMDB | Macro F1: 51.41 -> 52.38 | Helped |
| add I²MoE interaction experts and adaptive weighting to MulT | MulT · MOSI | Accuracy: 68.80 -> 71.91 | Helped |
| add the interaction loss (ablation) | I²MoE-MulT · ADNI | Accuracy: 58.73 (-6.35) | Helped |
| apply interaction loss to expert outputs rather than latent embeddings (ablation) | I²MoE-MulT · ADNI | Accuracy: 58.17 (-6.91) | Helped |
| use sample-specific reweighting rather than a shared global weight (ablation) | I²MoE-MulT · ADNI | Accuracy: 59.29 (-5.79) | Helped |
| perturb all modalities rather than mask only two per sample (ablation) | I²MoE-MulT · ADNI | Accuracy: 59.76 (-5.32) | Helped |
| include uniqueness experts alongside synergy and redundancy experts (ablation) | I²MoE-MulT · ADNI | Accuracy: 56.77 (-8.31) | Helped |
Surprising findings
- On MIMIC, I²MoE lowers accuracy when combined with SwitchGate, MoE++, and InterpretCC (up to -11.82 points for MoE++) while AUROC rises, which the authors attribute to dataset imbalance.
- Random-vector masking outperforms mean and zero vector replacement for generating interaction supervision on most datasets.
Limitations
- Accuracy drops on the imbalanced MIMIC dataset when combined with several fusion backbones.
- When experts disagree, accuracy is low on IMDB (15.85% correct) and ENRICO (46.85% correct), indicating difficulty handling disagreement.
- Computational overhead grows with the number of modalities (training roughly 2x or more on several datasets).
Experimental setup
Datasets: ADNI MIMIC-IV IMDB MOSI ENRICO
Benchmarks: ADNI MIMIC-IV IMDB MOSI ENRICO
Metrics: accuracy AUROC micro F1 macro F1
Baselines: EF LF LRMF MulT InterpretCC SwitchGate MoE++
Builds on: Partial Information Decomposition MulT Switch Transformer MoE++ InterpretCC MMoE
Closest papers by meaning
- InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions (NeurIPS 2025 · similarity 0.88)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.87)
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.87)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.87)
- Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.