Research map / Detection and segmentation / Open-set and few-shot detection
Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes universal medical image segmentation and Hermes, a context-prior learning method that learns task and modality prior tokens jointly with a segmentation backbone across 11 partially labeled datasets.
Problem
Medical image segmentation datasets are partially annotated, multi-modal and multi-region with conflicting class definitions, so task-specific models fail to exploit shared structure across tasks and modalities.
Approach
Learnable task and modality context-prior tokens, selected via task/modality oracles at inference and fused with image features through bi-directional cross-attention at multiple scales, let one model handle many partially labeled segmentation tasks; the priors are learned from data rather than from one-hot or CLIP embeddings.
Key results
- Hermes-M (MedFormer backbone) reaches 85.28% average Dice across the 11 upstream datasets, versus 83.39% for traditional MedFormer.
- Hermes-R improves tumor segmentation on LiTS and KiTS by 2.62% and 3.57% over ResUNet, and gains 8.17% on PET (AutoPET) under the universal paradigm.
- In transfer learning on MSD Pancreas with 1% of data, Hermes-R reaches 72.71 pancreas Dice versus 56.08 for ResUNet pretrained on AMOS CT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a universal model with task and modality context priors rather than a task-specific ResUNet | Hermes-R · LiTS T | Dice: 64.87 -> 67.49 | Helped |
| Train a universal model with task and modality context priors rather than a task-specific ResUNet | Hermes-R · KiTS T | Dice: 81.89 -> 85.46 | Helped |
| Train a universal model with task and modality context priors rather than a task-specific ResUNet | Hermes-R · AutoPET | Dice: 65.52 -> 73.69 | Helped |
| Add hierarchical task priors at multiple scales (ablation) | Hermes · Universal segmentation (11 datasets) | Average Dice: 81.72 -> 82.98 | Helped |
| Replace bidirectional attention with unidirectional cross-attention (ablation) | Hermes · Universal segmentation (11 datasets) | Average Dice: 82.98% to 82.04% | Hurt |
| Add modality priors (ablation) | Hermes · Universal segmentation (11 datasets) | Average Dice: 83.21 | Helped |
| Add auxiliary modality loss (ablation) | Hermes · Universal segmentation (11 datasets) | Average Dice: 83.21 -> 83.37 | Helped |
Surprising findings
- KiTS single-dataset pretraining lags behind training from scratch when full downstream data is used.
- CLIP text embeddings encode medical objects with uniformly high cosine similarity, whereas the learned task priors separate them meaningfully.
Limitations
- Performance may be improved by handling missing labels with self-supervised or semi-supervised learning, which is not explored.
- Integrating more kinds of medical prior knowledge at inference is left for future work.
Experimental setup
Datasets: BCV LiTS KiTS AMOS CT SegTHOR StructSeg (SS T, SS H) AMOS MR CHAOS M&Ms DLBS AutoPET MSD Pancreas
Benchmarks: BCV LiTS KiTS AMOS CT StructSeg SS T StructSeg SS H AMOS MR CHAOS M&Ms AutoPET DLBS MSD Pancreas
Metrics: Dice score
Baselines: nnUNet ResUNet SwinUNETR MedFormer DeSD UniMiSS Multi-Decoder DoDNet CLIP-Driven MultiTalent UniSeg
Builds on: DoDNet CLIP-driven universal model nnUNet MedFormer
Closest papers by meaning
- Show and Segment: Universal Medical Image Segmentation via In-Context Learning (CVPR 2025 · similarity 0.88)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.87)
- Tyche: Stochastic In-Context Learning for Medical Image Segmentation (CVPR 2024 · similarity 0.87)
- Knowledge Transfer Scaling Laws for 3D Medical Imaging (NeurIPS 2026 · similarity 0.87)
- S2S2: Semantic Stacking for Robust Semantic Segmentation in Medical Imaging (AAAI 2025 · similarity 0.86)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.