Research map / Vision-language and multimodal / Cross-modal learning
FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FuseMoE, a mixture-of-experts fusion transformer with a Laplace gating function for flexible multimodal EHR and other data with missing modalities and irregular sampling, backed by a convergence analysis of Laplace vs softmax gating.
Problem
Multimodal fusion struggles with a variable number of modalities, missing modalities, and irregularly sampled time series, especially in clinical EHR data.
Approach
Sparsely gated MoE fusion layer with per-modality routers, learnable missing-modality embeddings, an entropy regularizer for expert balance, mTAND-based irregularity encoding, and a Laplace (negative L2-distance) gating function claimed to give better parameter estimation rates than softmax gating.
Key results
- On MIMIC-IV 48-IHM, the Laplace-gated FuseMoE reaches 81.03 AUROC versus 78.76 for TFN and 79.65 for HAIM.
- On CMU-MOSEI, the joint experts and router variant reaches 86.41 Acc-2 and 85.43 F1, versus 84.97 F1 for Softmax-MoE.
- Adding CXR and ECG to FuseMoE improves 48-IHM AUROC to 83.87 (Laplace, three modalities) from 81.03 with vitals and notes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Laplace rather than Softmax gating (ablation) | FuseMoE · MIMIC-IV 25-PHE | AUROC: 70.54 $\pm$ 0.47 -> 71.23 $\pm$ 0.53 | Helped |
| use Laplace rather than Softmax gating (ablation) | MoE · CMU-MOSI | MAE: 0.69 $\pm$ 0.01 -> 0.65 $\pm$ 0.04 | Helped |
| use per-modality routers instead of a joint router (ablation) | FuseMoE · CMU-MOSI | Acc-2: 87.28 $\pm$ 0.35 -> 88.23 $\pm$ 0.57 | Helped |
| use per-modality routers instead of a joint router (ablation) | FuseMoE · CMU-MOSEI | Acc-2: 86.41 $\pm$ 0.36 -> 86.12 $\pm$ 0.19 | Mixed |
| add CXR to Vital & Notes (ablation) | FuseMoE with Laplace gating · MIMIC-IV 48-IHM | AUROC: 81.03 $\pm$ 0.25 -> 83.87 $\pm$ 0.33 | Helped |
| add CXR to Vital & Notes (ablation) | FuseMoE with Laplace gating · MIMIC-IV LOS | AUROC: 82.91 $\pm$ 1.02 -> 83.51 $\pm$ 0.51 | Helped |
| add CXR to Vital & Notes (ablation) | FuseMoE with Laplace gating · MIMIC-IV 25-PHE | AUROC: 71.23 $\pm$ 0.53 -> 73.65 $\pm$ 0.39 | Helped |
| add ECG to Vital & Notes & CXR (ablation) | FuseMoE with Laplace gating · MIMIC-IV 48-IHM | AUROC: 83.87 $\pm$ 0.33 -> 83.55 $\pm$ 0.49 | Hurt |
| add ECG to Vital & Notes & CXR (ablation) | FuseMoE with Laplace gating · MIMIC-IV LOS | AUROC: 83.51 $\pm$ 0.51 -> 83.58 $\pm$ 0.78 | No clear effect |
| add ECG to Vital & Notes & CXR (ablation) | FuseMoE with Laplace gating · MIMIC-IV 25-PHE | AUROC: 73.65 $\pm$ 0.39 -> 73.67 $\pm$ 0.71 | No clear effect |
| use per-modality or disjoint routers with entropy regularization instead of a joint router with importance loss (ablation) | FuseMoE on patients with missing modalities · MIMIC-IV 48-IHM and LOS | performance | Helped |
Surprising findings
- HAIM's F1 on 25-PHE (42.13) exceeds FuseMoE variants, which the authors attribute to XGBoost handling of missing minority classes.
- Adding CXR and ECG did not benefit HAIM, while FuseMoE gained from the extra modalities.
Limitations
- Encoding irregularities may cause over-parameterization when input size is small.
- Intermediate clinical notes were unavailable, so only radiology reports were used.
- Baselines were not designed to handle varying numbers of modalities, so adapting them required substantial changes.
Experimental setup
Datasets: MIMIC-III MIMIC-IV CMU-MOSI CMU-MOSEI PAM CIFAR-10
Benchmarks: MIMIC-IV 48-IHM MIMIC-IV LOS MIMIC-IV 25-PHE CMU-MOSI CMU-MOSEI PAM CIFAR-10
Metrics: AUROC F1 MAE Acc-2 Corr accuracy
Baselines: TFN MulT MAG HAIM MISTS Softmax-MoE GRU-D SeFT IP-Net Transformer
Builds on: sparsely gated mixture of experts multi-time attention (mTAND) Vision-MoE HAIM
Closest papers by meaning
- Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models (ICLR 2025 · similarity 0.87)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.86)
- R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025 · similarity 0.85)
- Predictive Dynamic Fusion (ICML 2024 · similarity 0.85)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.85)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.