Research map / Vision-language and multimodal / CLIP and zero-shot
Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP for chest X-ray report alignment by adding clinically-enhanced dynamic soft labels (text, CheXbert clinical, and RadGraph graph similarity), negation-based hard negatives, and introduces the CXR-Align benchmark.
Problem
General-domain CLIP-style contrastive training on medical image-report pairs struggles with negations and with semantic overlap/duplicate templated reports within batches, which creates false negatives and noise.
Approach
Replace one-hot InfoNCE targets with KL-divergence to thresholded, row-normalized soft labels derived from intra-modal similarities of report text embeddings, CheXbert clinical labels, and RadGraph graph embeddings, and add LLM-generated negated reports as hard negatives.
Key results
- The full model CLIP^{N,G}-D_{t+c+g} reaches 86.6% zero-shot RSNA accuracy and 63.0% CXR14 zero-shot accuracy, outperforming baselines such as BioViL-T (87.8 RSNA ZS is higher, so the gain is mainly in fine-tuned and other datasets).
- On adversarial prediction the model scores 34.4% accuracy versus 23.3% for the best baseline (BioViL).
- On CXR-Align task A, the hard-negative model reaches 96.5/94.0/96.4 on MIMIC/CheXpert/OpenI, while task B improves to 80.1/75.3/73.8 over CLIP's 72.4/70.3/62.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dynamic soft labels based on text and clinical similarity (ablation) | CLIP · RSNA ZS | AUC: 81.2 -> 84.4 | Helped |
| Add dynamic soft labels based on text and clinical similarity (ablation) | CLIP · SIIM ZS | AUC: 74.3 -> 84.8 | Helped |
| Train with negation-based hard negatives alone (ablation) | CLIP · RSNA ZS | AUC: 81.2 -> 82.0 | No clear effect |
| Combine negation-based hard negatives with dynamic soft labels (ablation) | CLIP · RSNA ZS | AUC: 84.4 -> 86.4 | Helped |
| Add graph embeddings and graph-based dynamic soft labels (ablation) | CLIP-D_{t+c} · CXR14 ZS | accuracy: 61.2 -> 62.3 | Helped |
| Add graph embeddings and graph-based dynamic soft labels (ablation) | CLIP^{N}-D_{t+c} · CXR14 ZS | accuracy: 62.8 -> 63.0 | Helped |
| Add negation-based hard negatives to CLIP-D_{t+c} under added normal-case imbalance (ablation) | CLIP-D_{t+c} · RSNA ZS with imbalanced training data | AUC: 82.1 -> 83.2 | Helped |
| Add clinical similarity to text-based dynamic soft labels (ablation) | CLIP-D_t · RSNA ZS with imbalanced training data | AUC: 78.8 -> 82.1 | Helped |
| Add negation-based hard negatives without dynamic soft labels (ablation) | CLIP · CXR-Align task B | accuracy: 72.4 -> 71.7 | Hurt |
| Add negation-based hard negatives (ablation) | CLIP · CXR-Align task A | accuracy: 75.4 -> 97.3 | Helped |
| Add dynamic soft labels and graph embeddings to negation-based hard-negative training (ablation) | CLIP^{N} · MIMIC CXR-Align task B | accuracy: 71.7 -> 80.1 | Helped |
| Add normal and duplicate reports to the training data (ablation) | CLIP^{N,G}-D_{t+c+g} · OpenI normal case detection | accuracy: 3.1 -> over 80% | Helped |
Surprising findings
- Negation-based hard negatives alone did not improve classification and degraded task-2 CXR-Align performance, but became beneficial when paired with dynamic soft labels.
- Adding 130,000 normal CXRs (increasing imbalance) reduced zero-shot accuracy but raised normal case detection accuracy above 80%, which the authors argue is necessary for needle-in-a-haystack retrieval.
Limitations
- CheXbert covers only 14 clinical labels, limiting clinical similarity coverage.
- Single-image scenario only; temporal (prior image) reasoning is left for future work.
- Hard-negative negation task may be solved via shortcuts exploiting unnatural negation phrasing.
Experimental setup
Datasets: MIMIC-CXR CheXpert OpenI RSNA Pneumonia SIIM Pneumothorax VinDr CXR14 private tertiary hospital dataset
Benchmarks: RSNA Pneumonia RSNA-ab SIIM VinDr CheXpert CXR14 CXR-Align OpenI normal case detection MIMIC report retrieval CheXpert report retrieval OpenI report retrieval
Metrics: accuracy AUC top-5 retrieval accuracy CheXbert F1 precision recall
Baselines: ConVIRT BioViL BioViL-T CXR-CLIP SoftCLIP CLIP GLORIA MedCLIP
Builds on: CLIP SoftCLIP InfoNCE BioViL-T ProbMed
Closest papers by meaning
- Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering (CVPR 2025 · similarity 0.87)
- KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · similarity 0.87)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.86)
- RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability (NeurIPS 2025 · similarity 0.86)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.86)
- Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.