Research map / Vision-language and multimodal / CLIP and zero-shot
Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAVL, a medical vision-language pre-training framework that decomposes disease names into LLM- and expert-curated visual aspects and uses a dual-head Transformer (supervised + contrastive) for zero-shot pathology detection on chest X-rays.
Problem
Medical VLP models like CheXzero and MedKLIP misalign chest X-ray images with complex report or disease-definition text, which hurts zero-shot detection of both seen and unseen (including rare) diseases.
Approach
Decompose each disease into a shared set of visual aspects (e.g., texture, shape, location) described by GPT-4 and curated by two medical experts, match images against each aspect, and pair a supervised Transformer head for seen diseases with a contrastive head for unseen diseases.
Key results
- Zero-shot seen-disease F1 improves over MedKLIP by up to 8.56% across 5 datasets.
- On unseen Covid-19 CXR-2, contrastive head reaches 83.86 AUC vs 76.28 for MedKLIP and 73.13 for CheXzero.
- Fine-tuning with 1% data on SIIM Pneumothorax classification beats MedKLIP by 7.38% (AUC 91.53 vs 85.24).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decompose disease descriptions into visual aspects and match images to those aspects | MAVL · COVID-19 CXR-2 | AUC: 76.28 -> 83.86 | Helped |
| use the contrastive head rather than the supervised head (ablation) | MAVL · COVID-19 CXR-2 | AUC: 77.85 -> 83.86 | Helped |
| use the contrastive head rather than the supervised head (ablation) | MAVL · ChestXray-14 | AUC: 73.57 -> 71.16 | Hurt |
| add supervised learning to the contrastive-only model (ablation) | MAVL · COVID-19 CXR-2 | AUC: 81.65 -> 83.86 | Helped |
| add supervised learning to the contrastive-only model (ablation) | MAVL · ChestXray-14 | AUC: 62.08 -> 71.16 | Helped |
| add contrastive learning to the supervised-only model (ablation) | MAVL · COVID-19 CXR-2 | AUC: 77.85 -> 78.74 | Helped |
| add contrastive learning to the supervised-only model (ablation) | MAVL · ChestXray-14 | AUC: 72.71 -> 73.57 | Helped |
| add texture as a visual aspect alongside the disease description (ablation) | MAVL · COVID-19 CXR-2 | AUC: +4.02% | Helped |
| add visual aspects to disease-description matching (ablation) | MAVL · zero-shot classification across reported datasets | accuracy | Helped |
| increase the number of visual aspects matched (ablation) | MAVL · zero-shot classification | AUC | Helped |
Surprising findings
- Contrastive learning was found non-optimal for fine-grained seen-disease classification, so the supervised head outperforms it on base categories.
- The supervised MedKLIP-style model struggles on many unseen PadChest diseases, while contrastive heads generalize better to novel classes.
Limitations
- Aspect descriptions require GPT-4 generation plus manual curation by two medical experts, which is semi-manual and costly to extend.
- Evaluated only on chest X-ray datasets with ResNet-50 backbone.
- Unseen-disease gains are sensitive to distribution shift (e.g., PadChest from Spain vs MIMIC from US).
Experimental setup
Datasets: MIMIC-CXR v2 ChestX-ray14 CheXpert PadChest RSNA Pneumonia SIIM-ACR Pneumothorax COVIDx CXR-2 COVID Rural
Benchmarks: CheXpert ChestX-ray14 PadChest RSNA Pneumonia SIIM-ACR Pneumothorax COVIDx CXR-2 COVID Rural
Metrics: AUC F1 accuracy Dice IoU
Baselines: ConVIRT GLoRIA BioViL BioViL-T CheXzero MedKLIP
Builds on: CLIP MedKLIP CheXzero BioViL ConVIRT
Closest papers by meaning
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.88)
- Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation (CVPR 2025 · similarity 0.88)
- CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment (CVPR 2024 · similarity 0.88)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.88)
- Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images (CVPR 2024 · similarity 0.87)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.