Research map / Vision-language and multimodal / Multimodal LLMs
Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Uni-Med, a medical generalist MLLM with a connector mixture-of-experts (CMoE) that mitigates multi-task tug-of-war at the vision-language connector, trained jointly on 12 datasets across six medical tasks.
Problem
Multi-task interference (the "tug-of-war") when training a unified medical multimodal LLM across diverse tasks, where prior work mostly applies MoE to the LLM and ignores the connector.
Approach
Replace the standard connector with a mixture of projection experts (CMoE) using a soft router that takes token-level visual features and learnable task-level tokens, so different tasks get different alignment weights; also provides gradient-based and parameter-statistics analyses of task interference.
Key results
- CMoE with soft router and token&task routing improves total average gain by 8.0% over MLP connector in the ablation (Table a).
- Uni-Med reports BLEU-1 of 82.12 on Slake-VQA and 58.07 on Path-VQA, outperforming LLaVA-Med (76.95 and 46.42) and other baselines in the overall comparison.
- On MIMIC-CXR report generation, Uni-Med achieves RadGraph-F1 13.98 vs 9.00 for XrayGPT, and on OrganSMNIST classification 78.07% accuracy vs 18.90% for RadFM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the shared MLP connector with CMoE using a soft router (ablation) | Uni-Med multi-task medical model · medical multi-task evaluation across five task groups | Total Δ: 8.0% | Helped |
| combine token-level and task-level information for routing (ablation) | CMoE soft router · medical multi-task evaluation | BLEU-1: 82.51 / 57.43; 81.52 / 57.75 | Mixed |
| replace the soft router with a hard router (ablation) | CMoE · medical multi-task evaluation | Total Δ: 0.0%; 8.0% | Hurt |
| use task-level soft routing instead of token-level soft routing (ablation) | CMoE · medical multi-task evaluation | Total Δ: 7.8% | Helped |
| replace compression-rate-4 projection with max pooling (ablation) | visual feature resampler · medical multi-task evaluation | Total Δ: -4.1% | Hurt |
| increase projection experts from 5 to 10 (ablation) | CMoE · medical multi-task evaluation | Total Δ: 8.0% -> 7.9% | No clear effect |
| increase LoRA rank from 8 to 16 (ablation) | CMoE soft-router model · medical multi-task evaluation | Total Δ: 8.0% -> 7.4% | No clear effect |
| use separate LoRA-MoE instead of standard LoRA with an MLP connector (ablation) | Uni-Med · VQA, REC, REG, RG, and CLS | Total Δ: -1.4% | Mixed |
| add CMoE soft routing to LoRA-MoE fine-tuning (ablation) | LoRA-MoE model · medical multi-task evaluation | Total Δ: -1.4% -> 5.7% | Helped |
| jointly train one model across tasks rather than use the strongest reported task-specific baseline | Uni-Med · Path-VQA | BLEU-1: 46.42 -> 58.07±0.32 | Helped |
| jointly train one model across tasks rather than use the strongest reported baseline | Uni-Med · MIMIC-CXR | RadGraph-F1: 9.00 -> 13.98±2.45 | Helped |
Surprising findings
- Standard LoRA-MoE in the LLM improves some tasks but degrades others (e.g., RG BLEU-1 drops 30%), indicating it does not solve tug-of-war alone.
- The hard (task-separated) router gives the largest gain on classification but overall total gain of 0.0%, showing separate connectors per task are not beneficial overall.
Limitations
- 3D medical volumes are reduced to 2D slices, losing information.
- Only LLaMA2-7B backbone and 12 datasets (140k samples) explored; gains in larger settings untested.
- Theoretical analysis of tug-of-war remains limited; interpretation relies on combining existing methods.
Experimental setup
Datasets: MedQA PubMedQA Slake-VQA Path-VQA MIMIC-CXR MPx-Single MedMNIST v2 (DermaMNIST, OrganSMNIST) SA-Med2D-20M Slake-REC Slake-REG SA-Med2D-REC SA-Med2D-REG
Benchmarks: Slake-VQA Path-VQA MIMIC-CXR MPx-Single DermaMNIST OrganSMNIST Slake-REC SA-Med2D-REC Slake-REG SA-Med2D-REG
Metrics: BLEU-1 BLEU-4 ROUGE-1 ROUGE-2 ROUGE-L METEOR RadGraph-F1 RadCliQ F1 IoU
Baselines: Med-Flamingo RadFM LLaVA-Med XrayGPT LoRA-MoE MLP connector
Builds on: LLaVA LoRA Mixture-of-Experts EVA-CLIP Med-PaLM M
Closest papers by meaning
- SMMILE: An expert-driven benchmark for multimodal medical in-context learning (NeurIPS 2025 · similarity 0.91)
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.90)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.90)
- GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.89)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.89)
- MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.