Research map / 3D vision / 3D generation
VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents VISTA3D, a unified 3D CT foundation model combining promptable automatic segmentation (127 classes), 3D interactive point-click segmentation with a click-based merge for correction, and zero-shot segmentation distilled from SAM via 3D supervoxels.
Problem
Existing 3D medical segmentation models either give accurate automatic segmentation for fixed classes without interactive correction or zero-shot ability, while interactive 2D/SAM-based approaches are too labor-intensive for 3D volumes; a unified model is needed that supports the full human-in-the-l
Approach
A shared SegResNet encoder with two heads: a promptable automatic head (learnable class embeddings indexed by class ID) and a 3D interactive point head; zero-shot ability is obtained by distilling SAM feature maps into 3D SLIC supervoxels used as training targets, and a four-stage training recipe balances the branches. A connected-component merge algorithm lets clicks correct automatic outputs wit
Key results
- Average Dice across 14 test datasets: VISTA3D auto+point 0.792 vs nnUNet 0.718 and Auto3Dseg 0.706 (VISTA3D auto 0.711).
- Zero-shot supervoxel distillation boosts VISTA3D zero-shot performance by about 50% per the authors, with state-of-the-art 3D zero-shot results versus MedSAM and SegVol on novel-class datasets.
- On WORD with full training data, VISTA3D reaches 0.875 Dice, the highest among the compared baselines; with one-shot finetuning on mouse micro-CT it reaches 0.926 vs 0.820 for Auto3DSeg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add one corrective point to VISTA3D automatic segmentation | VISTA3D · Supported-class test datasets | average Dice | Mixed |
| Add SAM-derived 3D supervoxels for training (ablation) | VISTA3D interactive branch · Zero-shot external datasets | zero-shot performance: 50% | Helped |
| Use randomly initialized class embeddings instead of CLIP embeddings (ablation) | VISTA3D automatic head · Supported-class segmentation | segmentation performance | Helped |
Surprising findings
- SAM2 finetuned on 3D medical data still underperforms VISTA3D, especially on complex 3D structures.
- CLIP text embeddings for class prompts gave slightly worse results than randomly initialized class embeddings.
Limitations
- Supports only 127 automatic classes and CT modality; MRI and PET not yet supported.
- Zero-shot performance still trails expert models on some datasets; single-click interactive results are below automatic on some tasks (e.g., AMOS22, spleen).
Experimental setup
Datasets: MSD BTCV VerSe AbdomenCT-1K AMOS22 TotalSegmentator dataset (TotalSegV2) Airway Bone Lesion (private) WORD Micro-CT mouse C4KC-KITS Adrenocortical Carcinoma
Benchmarks: MSD BTCV-Abdomen BTCV-Cervix VerSe AbdomenCT-1K AMOS22 TotalSegV2 Airway Bone Lesion Micro-CT mouse WORD C4KC-KITS
Metrics: Dice score inference time
Baselines: nnU-Net Auto3DSeg TotalSegmentator MedSAM SegVol SAM2-based finetuning SAM3D VISTA-NoSupervoxel
Builds on: Segment Anything (SAM) SegResNet nnU-Net FocalClick SLIC
Closest papers by meaning
- UniPro: Unified Multi-Mode Medical Image Segmentation from 2D Images to 3D Volumes via Propagation (NeurIPS 2026 · similarity 0.89)
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.87)
- Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation (NeurIPS 2024 · similarity 0.87)
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation (NeurIPS 2024 · similarity 0.86)
- EffiDec3D: An Optimized Decoder for High-Performance and Efficient 3D Medical Image Segmentation (CVPR 2025 · similarity 0.86)
- Integrating Deep Metric Learning with Coreset for Active Learning in 3D Segmentation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.