Research map / Detection and segmentation / Open-set and few-shot detection
"Why Is There a Tumor?": Tell Me the Reason, Show Me the Evidence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates a 180K image-mask-rationale prostate MRI dataset from PI-RADS reports via GPT-4o and a PI-RADS decision tree, and trains a MedSAM-based vision-language model with disentanglement and localization constraints to produce clinical-concept rationales grounded in visual evidence.
Problem
Medical segmentation and detection models output masks or labels without clinically meaningful rationales, while text-based explanations lack spatial grounding; paired image-mask-rationale data and pixel-level concept annotations are scarce.
Approach
Extract radiologist-style rationales from PI-RADS reports using GPT-4o guided by a structured PI-RADS decision tree, then train a shared image-text embedding with segmentation loss plus disentanglement and localization constraints so that concept heatmaps localize without pixel-level concept labels.
Key results
- Lesion AP 0.574 and AUROC 0.888 on the rationale dataset, outperforming U-Net and other baselines in cancer detection.
- Visual evidence RMA of 0.403 on the rationale dataset versus 0.010 for the model without constraints.
- Zero-shot on Prostate158 averages DSC 0.607 vs 0.558 for the best baseline (U-Net), with AP 0.565 vs 0.491.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add rationale-informed optimization with image-text rationales | Prostate tumor segmentation and detection model · Prostate158 zero-shot | Lesion DSC: 0.383 -> 0.426 | Helped |
| Add rationale-informed optimization with image-text rationales | Prostate tumor segmentation and detection model · Prostate158 zero-shot | AP: 0.491 -> 0.565 | Helped |
| Add rationale-informed optimization with image-text rationales | Prostate tumor segmentation and detection model · MSD-Prostate zero-shot | Zone DSC: 0.709 -> 0.717 | Helped |
| Add disentanglement and localization constraints (ablation) | Rationale-informed optimization · Rationale Dataset | Visual evidence RMA: 0.010 -> 0.403 | Helped |
| Add disentanglement and localization constraints (ablation) | Rationale-informed optimization · Prostate158 | Visual evidence RMA: 0.024 -> 0.330 | Helped |
| Add disentanglement and localization constraints (ablation) | Rationale-informed optimization · MSD-Prostate | Visual evidence RMA: 0.012 -> 0.497 | Helped |
| Add disentanglement and localization constraints (ablation) | Rationale-informed optimization · Rationale Dataset | Segmentation DSC: 0.758 -> 0.746 | Hurt |
| Add disentanglement constraint only (ablation) | Rationale-informed optimization · Rationale Dataset | Visual evidence RMA: 0.010 -> 0.024 | Helped |
| Add disentanglement constraint only (ablation) | Rationale-informed optimization · Rationale Dataset | Segmentation DSC: 0.758 -> 0.747 | Hurt |
| Add localization constraint to disentanglement constraint (ablation) | Rationale-informed optimization · Rationale Dataset | Visual evidence RMA: 0.024 -> 0.403 | Helped |
| Add localization constraint to disentanglement constraint (ablation) | Rationale-informed optimization · Rationale Dataset | Segmentation DSC: 0.747 -> 0.746 | Hurt |
Surprising findings
- OpenAI o1 performed worse than random guessing on tumor classification, location, and PI-RADS score prediction in the authors' 20-scan test.
- Using the disentanglement constraint alone led to trivial solutions with poor visual evidence localization (RMA 0.024).
Limitations
- Rationale dataset covers prostate cancer only.
- Lesion segmentation DSC on the rationale dataset is lower than some baselines (e.g., 0.552 vs 0.504 for U-Net on lesion but lower on gland), and the model is weaker on gland segmentation than U-Net and Swin UNETR.
- Explanations are not a substitute for radiologist judgment; over-reliance risk.
Experimental setup
Datasets: PI-CAI Prostate158 MSD-Prostate
Benchmarks: Rationale Dataset Prostate158 MSD-Prostate
Metrics: DSC AP AUROC RMA precision recall F1
Baselines: U-Net ITUNet Swin UNETR CSwin U-Net SSL CSwin U-Net MA-SAM SAMed BiomedParse OpenAI o1
Builds on: MedSAM BiomedCLIP InfoNCE contrastive learning Grad-CAM
Closest papers by meaning
- Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation (CVPR 2025 · similarity 0.86)
- TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis (ICLR 2026 · similarity 0.86)
- CG-Reasoner: Centroid-Guided Positional Reasoning Segmentation for Medical Imaging with a Robust Visual-Text Consistency Metric (CVPR 2026 · similarity 0.85)
- MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.85)
- CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation (NeurIPS 2024 · similarity 0.85)
- Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.