Research map / Vision-language and multimodal / Human and embodied modeling
CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAT, a query-based promptable segmentation model that jointly uses cropped 3D anatomical volume prompts and LLM-generated textual knowledge prompts to segment abdominal organs and tumors, including colon tumors across cancer stages.
Problem
Promptable medical segmentation with only textual or only visual prompts fails on diverse anomalies such as tumors that vary widely in shape, size, and appearance, and long-tailed data limits text-visual alignment.
Approach
Coordinates anatomical prompts (cropped 3D volumes of the same class from other scans) with knowledge-enriched long textual prompts via a ShareRefiner that uses soft attention for segmentation/text queries and hard Gumbel-softmax assignment for anatomical queries, plus a PromptRefer with attention masks linking prompt groups to segmentation queries and InfoNCE contrastive alignment.
Key results
- On FLARE22 organ segmentation, CAT reaches 86.8% average DSC, versus 88.4% for CT-SAM3D (which uses multi-round prompts) and 84.7% for Universal.
- On MSD tumor segmentation, CAT achieves 72.73% liver tumor and 49.67% pancreas tumor DSC, outperforming SegVol and ZePT on most subtypes.
- On the in-house colon tumor set, CAT averages 53.35% DSC versus 50.28% for SegVol, with 57.37% on Stage-IV tumors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| coordinate anatomical and textual prompts | CAT · FLARE22 organ segmentation | average DSC (%) | Mixed |
| coordinate anatomical and textual prompts | CAT · MSD tumor segmentation | DSC (%) | Mixed |
| remove anatomical and textual prompts (ablation) | CAT · Duodenum and liver tumor ablation | DSC (%): 64.08→54.39; 72.49→66.37 | Hurt |
| add anatomical prompts to a textual-prompt model (ablation) | CAT · Duodenum and liver tumor ablation | DSC (%): 57.31→64.08; 69.13→72.49 | Helped |
| add textual prompts to an anatomical-prompt model (ablation) | CAT · Duodenum and liver tumor ablation | DSC (%): 60.45→64.08; 64.82→72.49 | Helped |
| add anatomical prompts (ablation) | CAT · Esophagus and hepatic vessel tumor ablation | DSC (%): 72.81→78.99; 65.18→69.40 | Mixed |
| add textual prompts (ablation) | CAT · Pancreas tumor ablation | DSC (%): 45.08→44.55 | Helped |
| enable hard assignment for anatomical queries (ablation) | CAT ShareRefiner · organ and tumor ablation | DSC (%): 70.26→73.46 | Helped |
| apply hard assignment to textual and segmentation queries as well (ablation) | CAT ShareRefiner · organ and tumor ablation | DSC (%) | Hurt |
| remove the PromptRefer attention mask (ablation) | CAT PromptRefer · Stage-IV colon tumor | DSC (%): 58.49→56.41 | Hurt |
| remove contrastive alignment (ablation) | CAT · segmentation query feature separation | feature separation | Hurt |
Surprising findings
- Anatomical-prompt-only variant performs poorly on tumor segmentation, indicating visual prompts alone lack a general concept for diverse tumors.
- Applying hard assignment to all query types decreases performance, suggesting entanglement among prompt types.
Limitations
- Initial anatomical prompt embeddings are not well separated, reflecting the lack of general CT prompt encoders; CT foundation models needed.
- Errors may occur on rare lesion types or after radical resection that changes anatomy.
- In-house dataset is private, and some cancer stages could not be identified in some cases.
Experimental setup
Datasets: BTCV CT-ORG AbdomenCT-1K CHAOS AMOS22 WORD Pancreas-CT LiTS KiTS MSD FLARE22
Benchmarks: FLARE22 MSD in-house colon tumor dataset
Metrics: DSC HD95
Baselines: nnUNet Swin UNETR SAM MedSAM SAM-Med2D SAM-Med3D SegVol CT-SAM3D Universal ZePT
Builds on: Mask2Former CLIP Swin UNETR ZePT
Closest papers by meaning
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.87)
- Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model Using 3D Whole-Body CT Scans (AAAI 2025 · similarity 0.87)
- ZePT: Zero-Shot Pan-Tumor Segmentation via Query-Disentangling and Self-Prompting (CVPR 2024 · similarity 0.87)
- Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation in Medical Image Segmentation (AAAI 2025 · similarity 0.86)
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation (NeurIPS 2024 · similarity 0.86)
- One-Prompt to Segment All Medical Images (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.