Research map / Label-efficient and robust learning / Recognition and long-tail
Low-Rank Knowledge Decomposition for Medical Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoRKD, which decomposes a medical foundation model (ResNet50) into lightweight task-specific expert models (ShuffleNetV2 with low-rank expert modules and efficient knowledge separation convolution) that can be fused into single deployable networks.
Problem
Medical foundation models generalize well but underperform task-specific models on specific medical tasks and are costly to deploy; the goal is to split them into lightweight per-department experts that are specialized yet transferable.
Approach
Attach low-rank (LoRA-style) task-specific expert modules to each convolution of a shared backbone, and use an efficient knowledge separation convolution that aggregates per-sample expert weights so that gradients are separated per task in a single forward pass; after training, expert weights are fused so each expert deploys at the size of the backbone.
Key results
- On Radimagenet, LoRKD achieves 56.26% average decomposition accuracy with 2.21M training params, versus 51.01% for KF (5.01M) and 43.74% for the ResNet50 foundation model.
- On seven downstream datasets with Radimagenet pre-training, LoRKD (1.25M deployed params) averages 84.18% vs 80.66% for KF and 80.34% for the foundation model.
- Deployed LoRKD experts have 5.32% of the foundation model's parameters (1.25M vs 23.51M) and equal FLOPs to the baseline (0.15G vs 4.13G) when parameters are fused.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decompose the foundation model with LoRKD | medical foundation models · Radimagenet | average accuracy: 55.81 -> 56.26 | Helped |
| decompose the foundation model with LoRKD | medical foundation models · MedMnist | average accuracy: 83.33 -> 83.50 | Helped |
| decompose the foundation model with LoRKD | medical foundation models · Med-MT | average accuracy: 73.80 -> 75.73 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · COVID | accuracy: 83.25 -> 86.21 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · BTC | accuracy: 79.95 -> 81.47 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · AD | accuracy: 76.08 -> 79.12 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · Mura | accuracy: 76.73 -> 79.57 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · AUITD | accuracy: 97.77 -> 98.33 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · HAM10000 | accuracy: 75.83 -> 76.03 | Helped |
| use the Radimagenet-decomposed LoRKD expert | downstream models · DET10 | accuracy: 87.54 -> 88.50 | Helped |
| increase low-rank expert rank from 4 to 8 (ablation) | LoRKD · Radimagenet | average accuracy: 55.08 -> 56.26 | Helped |
| increase low-rank expert rank from 8 to 16 (ablation) | LoRKD · Radimagenet | average accuracy: 56.26 -> 56.19 | Hurt |
Surprising findings
- The unified-head foundation model is inferior to single-task STL models on Radimagenet and MedMnist, with MedMnist foundation accuracy only 50.05% average, suggesting specialization diminishes as pre-training diversity grows.
- Standard MTL outperforms STL on Radimagenet and Med-MT but underperforms on MedMnist, and advanced MTL methods (MoCo-MTL, Aligned-MTL) do not improve over standard MTL.
Limitations
- Foundation-model baseline uses a unified classification head, which the authors acknowledge makes its reported performance relatively poor.
- Advantage over KF on transferability depends on fine-tuning protocols; for downstream tasks without a matching expert, fine-tuning uses the shared backbone.
- Choice of rank r is not monotonic: r=16 does not further improve over r=8 on Radimagenet, suggesting dependence on dataset scale.
Experimental setup
Datasets: Radimagenet MedMnist Med-MT COVID BTC AD Mura AUITD HAM10000 DET10
Benchmarks: Radimagenet MedMnist Med-MT COVID BTC AD Mura AUITD HAM10000 DET10
Metrics: accuracy (task-level average) parameter count FLOPs MIG disentanglement score Grad-CAM visualization
Baselines: STL MTL STL-KD MTL-KD MoCo-MTL Aligned-MTL KF Foundation model (ResNet50)
Builds on: LoRA KF (knowledge factorization) Knowledge distillation FLoRA
Closest papers by meaning
- Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE (NeurIPS 2024 · similarity 0.84)
- A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis (NeurIPS 2024 · similarity 0.83)
- VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · similarity 0.83)
- A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models (AAAI 2025 · similarity 0.82)
- KIND: Knowledge Integration and Diversion for Training Decomposable Models (ICML 2025 · similarity 0.82)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.