Research map / Vision-language and multimodal / Cross-modal learning
Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Lego (MM-Lego), a wrapper (LegoBlock) that maps any unimodal encoder into a frequency-domain latent of shared shape, enabling training-free model merging (LegoMerge) and light fine-tuning fusion (LegoFuse) for biomedical multimodal tasks.
Problem
Multimodal fusion in biomedicine typically requires end-to-end training, scales poorly with number of modalities, handles modality imbalance and non-overlapping training sets poorly, and is topology-specific; existing model merging assumes identical architectures.
Approach
A LegoBlock wraps each unimodal encoder with an iterative cross-attention update in the frequency domain (keeping real and imaginary parts), giving all modalities a shared latent shape; merging is done via harmonic mean of magnitudes and arithmetic mean of phases, and task heads are merged via SLERP.
Key results
- LegoFuse (2 epochs) achieves the highest score on 5 of 7 datasets (e.g., BLCA c-Index 0.734, KIRP 0.863, ICD9 AUC 0.771) and is top-2 on all datasets.
- LegoMerge, with no multimodal training, beats the best unimodal LegoBlock in 6 of 7 datasets (e.g., ICD9 uplift 5.7%, MORT 5.3%) and is competitive with multimodal baselines.
- LegoMerge beats a naive SNN-AMIL logit ensemble by 7.1% (ICD9) and 7.3% (MORT) on MIMIC tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · BLCA | c-Index: 0.701 -> 0.734 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · BRCA | c-Index: 0.601 -> 0.626 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · KIRP | c-Index: 0.825 -> 0.863 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · UCEC | c-Index: 0.625 -> 0.634 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · MIMIC ICD9 | AUC: 0.684 -> 0.771 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · MIMIC MORT | Macro AUC: 0.751 -> 0.759 | Helped |
| Fine-tune the stacked LegoBlocks for 2 epochs instead of using the unfine-tuned LegoMerge | MM-Lego · ISIC | AUC: 0.721 -> 0.701 | Hurt |
| Use the weaved variant instead of the stacked variant (ablation) | LegoFuse · Seven biomedical datasets | task performance | No clear effect |
| Merge modality-specific LegoBlocks without multimodal fine-tuning | LegoMerge · Six of seven biomedical datasets | task performance: 2.9%; 1.7%; -1.8%; 1.6%; 5.7%; 5.3%; 2.1% | Helped |
Surprising findings
- LegoMerge without any multimodal training is competitive with end-to-end multimodal baselines on most datasets.
- Several end-to-end baselines (e.g., MultiModN, SNN+ABMIL bilinear on MIMIC) collapse to AUC 0.5 on some tasks.
Limitations
- Requires a small amount of supervised training samples to fit LegoBlock adapters for LegoMerge to be effective.
- High variance and instability on the small TCGA-KIRP dataset (n=284).
- Evaluated only on biomedical datasets with three modality types; generality to vision-language not demonstrated.
Experimental setup
Datasets: TCGA (BLCA, BRCA, KIRP, UCEC) MIMIC-III SIIM-ISIC
Benchmarks: TCGA survival (c-Index) on BLCA, BRCA, KIRP, UCEC MIMIC-III ICD9 (AUC) and MORT (macro AUC) SIIM-ISIC (AUC)
Metrics: c-Index AUC macro AUC
Baselines: SNN AMIL MultiModN Perceiver SNN+ABMIL (concat, late) SNN+ABMIL (bilinear, late) MCAT HEALNet
Builds on: model merging SLERP TIES DARE Perceiver MultiModN
Closest papers by meaning
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.88)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process (NeurIPS 2025 · similarity 0.86)
- Structure-Aware Fusion with Progressive Injection for Multimodal Molecular Representation Learning (NeurIPS 2025 · similarity 0.86)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.86)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.