Research map / Vision-language and multimodal / Cross-modal learning
Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AMC, a top-down dynamic multi-modal fusion method that scores modality importance from attention and replaces low-informative tokens with inter-modal tokens, evaluated on TCGA, ADNI, MIMIC-IV and a non-medical benchmark.
Problem
Fusing heterogeneous medical modalities whose data quality varies across modalities and patients (including modality batch effects), where existing token fusion requires explicit cross-modal alignment that medical data lacks.
Approach
Quantify each modality's importance from attention statistics, then prune the least informative tokens per modality (count proportional to importance) and replace them with tokens from other modalities selected via a two-level contrastive alignment, using differential attention and a sparse mixture-of-experts transformer block to make the attention signal reliable and reduce gradient conflict.
Key results
- On TCGA, AMC reaches the best average c-index (0.734 vs 0.711 for the next best, LiMoE) and the best result on UCEC, LUAD and BLCA.
- On MIMIC-IV, AMC achieves the best Recall (61.65), F1 (59.75) and Precision among most baselines, though accuracy (68.51) is lower than several baselines such as MAGGate (77.42).
- On ADNI, AMC improves accuracy to 55.91 versus 52.27 for FlexMoE, the next best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AMC instead of the strongest listed baseline | TCGA survival prediction · TCGA average | c-index: 0.711 -> 0.734 | Helped |
| Use AMC instead of the strongest listed baseline | ADNI diagnosis classification · ADNI | Acc: 52.27±2.99 -> 55.91±2.49 | Helped |
| Use AMC instead of the strongest listed baseline | ADNI diagnosis classification · ADNI | Precision: 58.72±0.91 -> 55.38±2.55 | Hurt |
| Use AMC instead of the strongest listed baseline | MIMIC-IV mortality prediction · MIMIC-IV | F1: 57.96±0.72 -> 59.75±1.00 | Helped |
| Add contrastive loss (ablation) | AMC · MIMIC-IV | F1: 57.71±0.90 -> 59.75±1.00 | Helped |
| Add differential attention (ablation) | AMC · MIMIC-IV | F1: 58.03±1.01 -> 59.75±1.00 | Helped |
| Add SMoE (ablation) | AMC · MIMIC-IV | F1: 57.71±0.79 -> 59.75±1.00 | Helped |
| Add token fusion (ablation) | AMC · MIMIC-IV | F1: 57.71±0.79 -> 59.75±1.00 | Helped |
| Use STD instead of Mean for modality importance (ablation) | AMC · MIMIC-IV | F1: 59.75±1.00 -> 59.53±1.21 | No clear effect |
| Use Max instead of Mean for modality importance (ablation) | AMC · MIMIC-IV | F1: 59.75±1.00 -> 56.66±6.52 | Hurt |
| Use original token-fusion scoring instead of attention-based token scores (ablation) | AMC · MIMIC-IV | F1: 59.75±1.00 -> 58.82±0.74 | Hurt |
| Use vanilla attention instead of differential attention (ablation) | AMC · MIMIC-IV | F1: 58.03±1.01 -> 59.75±1.00 | Hurt |
Surprising findings
- Several baselines (MAGGate, TF) collapse to predicting the majority class on MIMIC-IV, yielding higher accuracy but near-chance recall.
- Replacing the differential attention with vanilla attention for token scoring degrades performance, indicating attention over-allocation hurts modality importance estimates.
Limitations
- Accuracy is lower than several baselines on MIMIC-IV and ADNI precision, which the authors attribute to baseline majority-class reliance.
- Modality importance and token scores rely on attention maps whose interpretability is asserted rather than validated against clinical ground truth.
- Experiments use relatively small medical datasets and RTX 3090 GPUs, with no reported compute or runtime comparison.
Experimental setup
Datasets: TCGA MIMIC-IV ADNI
Benchmarks: TCGA ADNI MIMIC-IV
Metrics: c-index accuracy recall precision F1
Baselines: FlexMoE FuseMoE LiMoE MAGGate MulT Token Fusion MUSE
Builds on: token fusion differential transformer FlexMoE Q-Former
Closest papers by meaning
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.88)
- Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning (CVPR 2024 · similarity 0.88)
- Multi-modal Medical Diagnosis via Large-small Model Collaboration (CVPR 2025 · similarity 0.88)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.88)
- DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency (AAAI 2024 · similarity 0.87)
- HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.