Research map / Continual learning / Continual fine-tuning of LLMs
Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MedCoSS, a sequential, rehearsal-based continual self-supervised pre-training paradigm that assigns each medical modality (report, X-ray, CT, MRI, pathology) to its own stage to avoid modal data collision of joint SSL.
Problem
Joint self-supervised pre-training across unpaired medical modalities suffers from representation conflicts ("modal data collision") and high cost of integrating new modalities, while single-modal SSL fails to generalize across modalities and dimensions.
Approach
Treat multi-modal medical SSL as continual learning: pre-train sequentially one modality per stage, retain 5% of previous data via k-means-selected rehearsal buffer, and preserve knowledge with feature distillation (replacing the pretext task on buffer data) plus intra-modal mixup, using a dimension-agnostic ViT with per-dimension tokenizers.
Key results
- MedCoSS achieves best or second-best results on eight of nine downstream datasets, e.g., ChestXR ACC 94.31 vs 92.13 for pathology single-modal model.
- Feature distillation improves average classification ACC by 3.19% over vanilla ER in ablation.
- Sequential SSL without retention drops CXR-to-CT stage ACC from 95.0% to 89.7% on PudMed20k-related probe, while MedCoSS drops only from 95.3% to 94.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MedCoSS instead of CT pre-training | ViT/B medical SSL · RICORD (CT) | ACC: 80.56 -> 83.33 | Helped |
| Use MedCoSS instead of CT pre-training | ViT/B medical SSL · LiTS (CT) | Dice: 71.98 -> 72.01 | Helped |
| Use MedCoSS instead of MRI pre-training | ViT/B medical SSL · VS (MRI) | Dice: 89.90 -> 90.12 | Helped |
| Use MedCoSS instead of MRI pre-training | ViT/B medical SSL · LA (MRI) | Dice: 89.64 -> 90.46 | Helped |
| Use MedCoSS instead of Path. pre-training | ViT/B medical SSL · NCH (Path.) | ACC: 95.62 -> 95.76 | Helped |
| Use MedCoSS instead of single-modal pre-training | ViT/B medical SSL · ChestXR (X-ray) | ACC: 95.55 -> 94.31 | Hurt |
| Use MedCoSS instead of single-modal pre-training | ViT/B medical SSL · QaTa (X-ray) | Dice: 80.27 -> 78.98 | Hurt |
| Use MedCoSS instead of single-modal pre-training | ViT/B medical SSL · GlaS (Path.) | Dice: 89.51 -> 89.13 | Hurt |
| Add feature distillation to ER with random sampling (ablation) | ER continual SSL · Average classification tasks | ACC: +3.19% | Helped |
| Add feature distillation to ER with random sampling (ablation) | ER continual SSL · Average segmentation tasks | Dice: +1.47% | Helped |
| Replace random sampling with k-means sampling (ablation) | ER continual SSL · Five downstream datasets | generalization performance | No clear effect |
| Combine k-means sampling with feature distillation (ablation) | ER continual SSL · Five downstream datasets | generalization performance | Helped |
Surprising findings
- Joint multi-modal SSL underperforms single-modal pre-training on identical-modality downstream tasks, revealing modal data collision.
- Increasing the rehearsal buffer from 5% to 10% degrades MedCoSS on several datasets.
Limitations
- Paper acknowledges MedCoSS may not achieve state-of-the-art on all tasks (underperforms on PudMed20k).
- Modality order affects results and buffer size beyond 5% can hurt performance.
- Evaluated only in a single pre-training order for main results; continual streams with multiple modality sets left to future work.
Experimental setup
Datasets: MIMIC-CXR DeepLesion ADNI TCGA PubMed20k ChestXR QaTa-COV19 RICORD LiTS VS LA NCH
Benchmarks: PubMed20k ChestXR QaTa-COV19 RICORD LiTS VS LA NCH GlaS
Metrics: ACC AUC F1 Dice HD95
Baselines: Single-modal SSL (Report, X-ray, CT, MRI, Path.) Joint SSL with dimension-shared decoders Joint SSL with modal-specific decoders EWC ER PackNet CaSSLe TFS
Builds on: DER masked autoencoder BERT CaSSLe
Closest papers by meaning
- Multi-modal Vision Pre-training for Medical Image Analysis (CVPR 2025 · similarity 0.88)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.88)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.86)
- Advancing Multiple Instance Learning with Continual Learning for Whole Slide Imaging (CVPR 2025 · similarity 0.85)
- Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning (AAAI 2025 · similarity 0.85)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.