Research map / Vision-language and multimodal / CLIP and zero-shot
Unleashing the Potential of Vision-Language Pre-Training for 3D Zero-Shot Lesion Segmentation via Mask-Attribute Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Malenia aligns multi-scale lesion mask tokens with structured disease-attribute text embeddings via multi-positive contrastive loss and a cross-modal knowledge injection module for 3D zero-shot lesion segmentation on CT.
Problem
Zero-shot segmentation of unseen lesions in 3D CT scans is hard because image-level vision-language pre-training does not transfer to pixel-level lesion segmentation, and lesions vary widely in shape, size and boundaries.
Approach
Decompose radiology reports into eight shared visual attribute descriptions (location, shape, density, etc.), align multi-scale mask-token embeddings with all positive attribute embeddings using a multi-positive NCE loss, and fuse mask and text embeddings via cross-attention (CMKI) before ensembling their mask predictions.
Key results
- On unseen lesions, Malenia reaches 59.52% DSC on hepatic vessel tumor and 54.96% on kidney tumor, beating the best prior method ZePT (53.12% and 46.82%).
- In the fine-tuning setting, Malenia beats CT-GLIP by 6.03% DSC on hepatic vessel tumor.
- Ablations show multi-scale alignment, multi-positive contrastive loss and deep fusion each add gains; using both mask and text predictions beats either alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Malenia rather than the strongest listed baseline, ZePT | Zero-shot lesion segmentation · MSD, KiTS23, and in-house unseen lesion categories | DSC: at least a 6.40% improvement in DSC on MSD, a 8.14% improvement in DSC on KiTS23, and a 9.08% improvement in DSC on the | Helped |
| Use Malenia rather than supervised segmentation baselines | Seen-lesion segmentation · MSD tumors and KiTS23 kidney cysts | DSC: 1.46%; 1.35% | Helped |
| Add structured visual-attribute descriptions to raw-report alignment (S1) (ablation) | Malenia mask-attribute alignment · MSD Pancreas Tumor; KiTS23 Kidney Tumor; in-house Gallbladder Tumor | DSC: 38.77±10.30 -> 39.83±10.13; 47.05 -> 48.69; 36.12 -> 38.36 | Helped |
| Use multi-scale features for cross-modal alignment (S2) (ablation) | Malenia mask-attribute alignment · MSD Pancreas Tumor; KiTS23 Kidney Tumor; in-house Gallbladder Tumor | DSC: 38.77±10.30 -> 40.46±9.95; 47.05 -> 49.63; 36.12 -> 39.74 | Helped |
| Use multi-positive attribute contrastive learning (S3) with structured attributes (ablation) | Malenia mask-attribute alignment · MSD Pancreas Tumor; KiTS23 Kidney Tumor; in-house Gallbladder Tumor | DSC: 41.88±9.88 -> 42.23±9.71; 51.85 -> 53.54; 42.53 -> 44.40 | Helped |
| Add deep fusion to prediction using text embeddings and mask tokens (ablation) | Malenia CMKI · MSD Pancreas Tumor; KiTS23 Kidney Tumor; in-house Gallbladder Tumor | DSC: 41.22±9.90 -> 43.30±9.29; 52.58 -> 54.96; 43.07 -> 47.35 | Helped |
| Use both text embeddings and mask tokens rather than either modality alone (ablation) | Malenia CMKI prediction · MSD Pancreas Tumor; KiTS23 Kidney Tumor; in-house Gallbladder Tumor | DSC: with deep fusion: 40.50±9.93 and 42.44±9.48 -> 43.30±9.29; without deep fusion: 38.96±10.22 and 40.27±10.01 -> 41.22±9.9 | Helped |
| Switch attribute extraction from GPT-4 to MMed-Llama-3-8B (ablation) | Malenia in-house attribute construction · In-house Liver Cyst, Kidney Stone, and Gallbladder Tumor | DSC: 54.98 vs 54.97; 38.67 vs 38.66; 43.28 vs 43.28 | No clear effect |
| Add human annotator review and supplementation to LLM-extracted attributes (ablation) | Malenia in-house attribute construction · In-house unseen lesions | DSC | Helped |
| Increase the number of attribute aspects from 1 to 8 (ablation) | Malenia mask-attribute alignment · Unseen lesion categories | segmentation performance | Helped |
| Increase mask-token count beyond 16 (ablation) | Malenia · KiTS23 and in-house unseen lesions | segmentation performance | No clear effect |
Surprising findings
- Using GPT-4 or the open MMed-Llama-3-8B for attribute extraction gives nearly identical results, while human radiologist annotation adds a large gain.
Limitations
- The private in-house dataset is small (30 cases per lesion type) and not released.
- Attribute annotation relies on two radiologists supplementing reports, which is costly.
- Lesion-attribute matching is weaker for attributes such as surface characteristics, density and specific features.
Experimental setup
Datasets: MSD KiTS23 in-house CT dataset (China Medical University First Hospital)
Benchmarks: MSD KiTS23 in-house dataset
Metrics: DSC NSD
Baselines: SAM SAM 2 SaLIP H-SAM ZePT TransUNet nnUNet Swin UNETR Universal Model GLoRIA BioViL MedKLIP
Builds on: Mask2Former CLIP nnUNet MP-NCE
Closest papers by meaning
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.87)
- EasyLens: A Training-Free Plug-and-Play Subtle-Lesion Representation Amplifier for Medical Vision-Language Models (NeurIPS 2026 · similarity 0.86)
- CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment (CVPR 2024 · similarity 0.85)
- Why Invariance is Not Enough for Biomedical Domain Generalization and How to Fix It (NeurIPS 2026 · similarity 0.85)
- XMask3D: Cross-modal Mask Reasoning for Open Vocabulary 3D Semantic Segmentation (NeurIPS 2024 · similarity 0.85)
- Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.