Research map / Vision-language and multimodal / Cross-modal learning
Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Flex-MoE handles arbitrary missing-modality combinations with a learnable missing-modality bank indexed by observed combinations and a sparse MoE trained first on full-modality samples (G-Router) then specialized per modality combination (S-Router) with a top-1 cross-entropy loss.
Problem
Multimodal learning with missing modalities: existing frameworks rely on single modalities or complete data and cannot flexibly model arbitrary observed modality combinations, as in Alzheimer's disease staging and MIMIC-IV mortality prediction.
Approach
A missing-modality bank provides learnable embeddings for absent modalities conditioned on the observed combination; an SMoE layer with one expert per modality combination is trained first on full-modality samples (generalized router) and then specialized by forcing the top-1 gate to the observed combination's expert via a cross-entropy loss (specialized router), with curriculum ordering of sample
Key results
- On ADNI with all four modalities, Flex-MoE reaches 66.11% ACC versus 59.52% for FuseMoE and 61.44% for MAG.
- On MIMIC-IV with lab, notes and ICD-9 codes, Flex-MoE reaches 76.81% ACC versus 64.62% for the best baseline (MulT).
- Ablation: removing both expert specialization and generalization drops ACC from 66.11 to 62.49 on ADNI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Flex-MoE instead of MAG | ADNI full-modality classification · ADNI, IGC B (full modalities) | ACC: 61.44 {$\pm{1.61}$} -> 66.11 ±{1.14} | Helped |
| Use Flex-MoE instead of MulT | MIMIC-IV all-modality mortality classification · MIMIC-IV, LNC | ACC: {64.62} {$\pm{0.44}$} -> 76.81 {$\pm{0.90}$} | Helped |
| Add expert specialization (ablation) | Flex-MoE · ADNI, IGC B (full modalities) | ACC: 62.75 -> 66.11 | Helped |
| Add expert specialization and expert generalization (ablation) | Flex-MoE · ADNI, IGC B (full modalities) | ACC: 62.49 -> 66.11 | Helped |
| Add the embedding bank (ablation) | Flex-MoE · ADNI, IGC B (full modalities) | ACC: 63.87 -> 66.11 | Helped |
| Sort samples in descending rather than random order (ablation) | Flex-MoE · ADNI, IGC B (full modalities) | ACC: 62.65 -> 66.11 | Helped |
| Sort samples in descending rather than ascending order (ablation) | Flex-MoE · ADNI, IGC B (full modalities) | ACC: 63.87 -> 66.11 | Helped |
| Increase the number of experts beyond 16 (ablation) | Flex-MoE · ADNI, full modalities | accuracy | No clear effect |
| Use more than one SMoE layer or add a Transformer block (ablation) | Flex-MoE · ADNI, full modalities | accuracy | Hurt |
| Use top-4 rather than top-2 expert selection (ablation) | Flex-MoE · ADNI, full modalities | accuracy | Helped |
Surprising findings
- FuseMoE performs better with three modalities than with all four on ADNI, indicating baselines do not exploit added modalities.
- Top-4 routing was found more effective than the commonly used top-2 in this setting.
Limitations
- Effectiveness depends on availability of comprehensive, high-quality patient data.
- Mean imputation for initial missing values in ADNI; zero-padding used for some baselines.
- Evaluated only on ADNI and MIMIC-IV with a small number of modalities.
Experimental setup
Datasets: ADNI MIMIC-IV
Benchmarks: ADNI MIMIC-IV
Metrics: ACC Macro F1 AUC
Baselines: FuseMoE MAG TF MulT LIMoE ShaSpec mmFormer
Builds on: sparse mixture-of-experts FuseMoE curriculum learning mean imputation
Closest papers by meaning
- Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.90)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.88)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.88)
- FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning (NeurIPS 2026 · similarity 0.88)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.