Research map / Detection and segmentation / Open-set and few-shot detection
SegVol: Universal and Interactive Volumetric Medical Image Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SegVol, a 3D foundation model for universal, interactive volumetric CT segmentation of 200+ anatomical categories using point, bbox, and text prompts plus a zoom-out-zoom-in inference mechanism.
Problem
Volumetric medical image segmentation lacks a universal 3D foundation model that covers many anatomical categories with easy user interaction (spatial and semantic prompts) and efficient inference on full 3D volumes.
Approach
A SAM-style 3D promptable model with a 3D ViT image encoder pretrained via SimMIM on 96K CTs, a frozen CLIP text encoder for semantic prompts, pseudo masks from Felzenszwalb-Huttenlocher to cover unlabeled categories, and a zoom-out-zoom-in inference that segments a coarse global mask then refines the ROI with sliding windows.
Key results
- On 22 tasks, SegVol outperforms competitors in 19 tasks; average Dice on AMOS22 validation is 0.8593 vs 0.6668 for the runner-up SAM-MED2D.
- Zoom-out-zoom-in achieves average Dice 0.7298 at 190 ms per case vs 0.6529 at 3331 ms for sliding window on the AMOS22 split.
- SegVol average lesion Dice exceeds nnU-Net by 14.76% across lung tumor, colon cancer, and liver tumor tasks (0.7122/0.6965/0.7825 vs 0.5963/0.3769/0.3606).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SegVol with bbox+text prompts instead of SAM-MED2D | Interactive segmentation methods · AMOS22 | average Dice: 0.6668 -> 0.8593; 19.25% | Helped |
| Use SegVol with bbox+text prompts instead of SAM-MED3D | Interactive segmentation methods · SegTHOR | average Dice: 0.7727 -> 0.8155; 4.28% | Helped |
| Replace sliding-window inference with zoom-out-zoom-in inference (ablation) | SegVol · AMOS22 15-organ test split | average Dice: 0.6529 -> 0.7298 | Helped |
| Replace resize inference with zoom-out-zoom-in inference (ablation) | SegVol · AMOS22 15-organ test split | average Dice: 0.4509 -> 0.7298 | Helped |
| Add spatial bbox prompts to text prompts (ablation) | SegVol · 19 segmentation tasks | average Dice: 5.85% | Helped |
| Add semantic text prompts to point prompts (ablation) | SegVol · 19 segmentation tasks | average Dice: 4.62% | Helped |
| Scale training data from one dataset to more datasets (ablation) | SegVol · BTCV 20% test split | Dice | Helped |
| Train SegVol on the full dataset collection instead of a single task dataset | SegVol · Lung Tumor | average Dice: 0.5963 -> 0.7122 | Helped |
Surprising findings
- Trained only on CT, SegVol reportedly generalizes to MRI segmentation with semantic and spatial prompts.
- A single SAM-style model jointly trained on 25 datasets outperforms task-specific nnU-Net on lesion tasks.
Limitations
- Gap remains relative to referring volumetric segmentation with free-form language.
- Text-prompt results are weaker on ULS23 due to unclear category naming.
- Trained mainly on CT; multi-modality joint training left to future work.
Experimental setup
Datasets: CHAOS HaN-Seg AMOS22 AbdomenCT-1k KiTS23 KiPA22 KiTS19 BTCV Pancreas-CT 3D-IRCADB FLARE22 TotalSegmentator
Benchmarks: AMOS22 ULS23 SegTHOR BTCV MSD FLARE22
Metrics: Dice score inference time per case MACs
Baselines: SAM MedSAM SAM-MED2D SAM-MED3D 3DUX-Net SwinUNETR nnU-Net
Builds on: Segment Anything (SAM) SimMIM CLIP SAM-Med3D
Closest papers by meaning
- UniPro: Unified Multi-Mode Medical Image Segmentation from 2D Images to 3D Volumes via Propagation (NeurIPS 2026 · similarity 0.87)
- VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging (CVPR 2025 · similarity 0.86)
- CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation (NeurIPS 2024 · similarity 0.86)
- Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation (NeurIPS 2024 · similarity 0.85)
- VOILA: Complexity-Aware Universal Segmentation of CT Images by Voxel Interacting with Language (AAAI 2025 · similarity 0.85)
- Interactive Medical Image Segmentation: A Benchmark Dataset and Baseline (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.