Research map / Detection and segmentation / Open-set and few-shot detection
Show and Segment: Universal Medical Image Segmentation via In-Context Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Iris, a 3D in-context medical image segmentation framework that distills reference image-label pairs into compact task embeddings, enabling segmentation of unseen classes and domains without fine-tuning.
Problem
Medical image segmentation generalizes poorly to out-of-distribution data and unseen classes, and existing in-context learning methods are 2D-based, underperform task-specific models, and recompute reference encodings per query.
Approach
A lightweight task encoding module decouples reference-based task encoding from query inference, producing reusable task embeddings (high-resolution foreground pooling plus learnable query tokens) that drive a query-based 3D mask decoder for single-pass multi-class segmentation; it also supports context ensemble, object-level retrieval, and in-context tuning.
Key results
- Iris reaches 84.52% average Dice on 12 in-distribution datasets, comparable to task-specific nnUNet (88.67 avg is nnUNet's reported row; Iris is compared against it) and above in-context baselines (best Tyche-IS 61.20%).
- On held-out OOD sets, Iris averages 69.03% across the table's columns versus best in-context baseline Tyche-IS 61.92%, with ACDC 86.45% and SegTHOR 82.77%.
- On novel classes with one reference, Iris reaches 28.28% Dice on MSD Pancreas Tumor and 69.03% on Pelvic, versus best competitor 11.97% and 61.92%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Iris for in-distribution segmentation instead of Multi-Talent | Iris · In-distribution, 12 datasets (AVG) | mean Dice (%): 84.47 -> 84.52 | Helped |
| Use Iris instead of the strongest listed competitor | Iris · MSD Pancreas (Tumor), unseen class | mean Dice (%): 11.97 -> 28.28 | Helped |
| Use Iris instead of the strongest listed competitor | Iris · Pelvic (Bone), unseen class | mean Dice (%): 61.92 -> 69.03 | Helped |
| Add foreground feature pooling (ablation) | Iris · In-dist (Small) | mean Dice (%): 62.13 -> 80.36 | Helped |
| Add query-based contextual encoding (ablation) | Iris · In-dist (Avg) | mean Dice (%): 82.47 -> 84.52 | Helped |
| Add high-resolution feature processing (ablation) | Iris · In-dist (Small) | mean Dice (%): 77.53 -> 80.36 | Helped |
| Add all three components: high-resolution processing, foreground pooling, and query encoding (ablation) | Iris · In-dist (Avg) | mean Dice (%): 82.10 -> 84.52 | Helped |
| Use object-level context retrieval instead of context ensemble with fewer references | Iris · OOD Dice, top 10-20% references | OOD Dice (%) | Helped |
| Increase the context ensemble reference pool | Iris · OOD Dice | OOD Dice (%) | Helped |
| Optimize task embeddings using in-context tuning with limited samples | Iris · OOD Dice | OOD Dice (%) | Hurt |
Surprising findings
- Iris's task embeddings cluster anatomical structures across datasets and modalities (e.g., IVC with portal/splenic veins) without anatomical supervision.
- Inference on 10 query volumes with 15 classes takes 2.0 s on one A100 for Iris, versus 1030 s for UniverSeg-128.
Limitations
- Performance gap remains with supervised upper bounds in some scenarios.
- Out-of-distribution generalization depends on the diversity of training tasks, requiring automated task creation.
- Low Dice on novel classes (e.g., 28.28% on MSD Pancreas Tumor) indicates limited adaptation quality.
Experimental setup
Datasets: AMOS BCV Brain CT/MR CHAOS KiTS LiTS MnM StructSeg CSI-Wat PET Tumor H&N Tho
Benchmarks: AMOS BCV Brain Tumor CHAOS KiTS LiTS MnM StructSeg CSI-Wat ACDC SegTHOR CSI variants
Metrics: Dice score inference time memory parameters
Baselines: nnUNet CLIP-driven model UniSeg Multi-Talent SAM SAM-Med2D SAM-Med3D SegGPT UniverSeg Tyche-IS
Builds on: UniverSeg Tyche SegGPT Mask2Former
Closest papers by meaning
- Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation (CVPR 2024 · similarity 0.88)
- UniPro: Unified Multi-Mode Medical Image Segmentation from 2D Images to 3D Volumes via Propagation (NeurIPS 2026 · similarity 0.86)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.85)
- Pancakes: Consistent Multi-Protocol Image Segmentation Across Biomedical Domains (NeurIPS 2025 · similarity 0.85)
- Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation (CVPR 2024 · similarity 0.85)
- Tyche: Stochastic In-Context Learning for Medical Image Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.