Research map / Detection and segmentation / Open-set and few-shot detection
ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZePT, a two-stage MaskFormer-based framework that splits object queries into fundamental (organ) and advanced (tumor) sets, uses self-generated anomaly-map prompts, and aligns queries with GPT-4-generated clinical text for zero-shot pan-tumor segmentation.
Problem
Zero-shot segmentation of unseen tumor categories in medical CT, where open-vocabulary segmentation proposals poorly cover subtle tumor regions and rare tumor types lack annotations.
Approach
Disentangle object queries into organ-level fundamental queries (trained first with object-aware feature grouping) and tumor-level advanced queries that attend to anomaly-score prompts derived from fundamental-query responses, plus feature-level query-knowledge alignment with ClinicalBERT embeddings of LLM-generated descriptions, avoiding a CLIP-based proposal classifier.
Key results
- On unseen tumors on MSD, ZePT improves average DSC by 15.85%, AUROC by 17.43%, and reduces FPR95 by 23.27% over prior OVSS methods.
- On seen-tumor segmentation, ZePT outperforms nnUNet and Swin UNETR by at least 4.83% DSC on BTCV, 4.51% on LiTS liver tumors, and 2.21% on KiTS.
- ZePT has 745.94M parameters and 1337.59G FLOPs versus 950.82M/5766.21G for ZegFormer-style OVSS baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ZePT instead of OVSeg | zero-shot tumor segmentation · MSD pancreas tumor | DSC: 19.36 -> 37.67 | Helped |
| use ZePT instead of OVSeg | zero-shot tumor segmentation · MSD lung tumor | DSC: 14.11 -> 27.22 | Helped |
| use ZePT instead of OVSeg | zero-shot tumor segmentation · MSD hepatic vessel tumor | DSC: 35.66 -> 52.94 | Helped |
| use ZePT instead of OVSeg | zero-shot tumor segmentation · MSD colon tumor | DSC: 15.76 -> 30.45 | Helped |
| use ZePT instead of OVSeg | zero-shot tumor segmentation · Real-world colon tumor | DSC: 16.05 -> 36.23 | Helped |
| add object-aware feature grouping (ablation) | ZePT · Real-world colon tumor | DSC: 31.28 -> 36.23 | Helped |
| use the Query Refinement Decoder instead of a vanilla transformer decoder (ablation) | ZePT · Real-world colon tumor | DSC: 25.84 -> 36.23 | Helped |
| use prompt-based masked attention instead of vanilla self-attention with concatenated visual prompts (ablation) | ZePT · Real-world colon tumor | DSC: 33.06 -> 36.23 | Helped |
| use query-knowledge alignment instead of CLIP image-encoder classification (ablation) | ZePT · Real-world colon tumor | DSC: 31.04 -> 36.23 | Helped |
| use self-generated visual prompts instead of relaxed 3d bounding-box prompts | ZePT · Real-world colon tumor | DSC: 35.89 -> 36.23 | Helped |
Limitations
- Zero-shot performance on unseen tumors still lags behind supervised models trained on those tumors.
- Real-world colon tumor dataset is private, limiting reproducibility.
- Knowledge descriptions were generated by GPT-4 and checked by doctors, requiring expert effort.
Experimental setup
Datasets: Pancreas-CT AbdomenCT-1K CT-ORG CHAOS AMOS22 BTCV WORD TotalSegmentator LiTS KiTS
Benchmarks: MSD Real-World Colon Tumor (private) BTCV LiTS KiTS
Metrics: DSC AUROC FPR95
Baselines: ZegFormer zsseg OpenSeg OVSeg FreeSeg SynthCP SML MaxQuery nnUNet Swin UNETR Universal model MedSAM
Builds on: MaskFormer Swin UNETR ZegFormer MaxQuery
Closest papers by meaning
- Advancing Generalizable Tumor Segmentation with Anomaly-Aware Open-Vocabulary Attention Maps and Frozen Foundation Diffusion Models (CVPR 2025 · similarity 0.89)
- CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation (NeurIPS 2024 · similarity 0.87)
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.85)
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.85)
- Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities (AAAI 2025 · similarity 0.85)
- Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.