Research map / Vision-language and multimodal / CLIP and zero-shot
MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLIP, a medical image-report pre-training framework combining a manually updated divergence encoder, UMLS-knowledge-guided local token-patch alignment, and knowledge-guided prototype clustering for category-level contrastive learning.
Problem
Medical visual representation learning from image-report pairs overlooks multi-granularity correspondences and suffers from false negatives in contrastive learning, underutilizing image-text data when annotations are scarce.
Approach
Uses two manually (momentum-style, cosine-weighted) updated divergence encoders as augmentation to generate diverse features, aligns image patches and report tokens through UMLS knowledge graph entities as an intermediate medium, and performs Sinkhorn-based prototype clustering so semantically similar samples across batches are treated as positives.
Key results
- On SIIM segmentation with 1% data, MLIP reaches 40.2 Dice vs 38.6 for PRIOR and 33.5 for MGCA (per Table 2); on RSNA segmentation with 1% data it outperforms MGCA by 4.7 points.
- Object detection on RSNA with 1% data: MLIP 17.2 mAP vs 15.6 for PRIOR; MLIP surpasses prior methods across zero-shot, 1%, 10%, and 100% settings in most cases.
- Classification with ViT-B/16 on RSNA AUC at 1% data: 89.3 for MLIP vs 89.1 for MGCA on CheXpert 1% (89.0 vs 88.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MLIP instead of PRIOR | ResNet-50 detection pre-training · RSNA Pneumonia | mAP: 25.2 -> 25.8 | Helped |
| Use MLIP instead of PRIOR | ResNet-50 detection pre-training · Object CXR | mAP: 19.8 -> 20.2 | Helped |
| Use MLIP instead of PRIOR | ResNet-50 segmentation pre-training · RSNA Pneumonia | Dice: 72.7 -> 73.5 | Helped |
| Use MLIP instead of PRIOR | ResNet-50 segmentation pre-training · SIIM Pneumothorax | Dice: 66.3 -> 68.1 | Helped |
| Use MLIP instead of MGCA | ViT-B/16 classification pre-training · CheXpert | AUC: 89.7 -> 90.0 | Helped |
| Use MLIP instead of MGCA | ViT-B/16 classification pre-training · RSNA Pneumonia | AUC: 90.8 -> 90.8 | No clear effect |
| Use MLIP instead of MGCA | ViT-B/16 classification pre-training · COVIDx | accuracy: 92.3 -> 92.5 | Helped |
| Add category-level ITA (ablation) | MLIP · RSNA Pneumonia | Dice: 71.7 -> 73.5 | Helped |
| Add category-level ITA (ablation) | MLIP · SIIM Pneumothorax | Dice: 64.6 -> 68.1 | Helped |
| Add local ITA (ablation) | MLIP · RSNA Pneumonia | Dice: 70.4 -> 73.5 | Helped |
| Add local ITA (ablation) | MLIP · SIIM Pneumothorax | Dice: 66.4 -> 68.1 | Helped |
| Add global ITA (ablation) | MLIP · RSNA Pneumonia | Dice: 73.3 -> 73.5 | Helped |
Surprising findings
- Using ViT-B/16, MLIP only comparably matches MGCA on CheXpert with 100% data, which the authors note as an exception to consistent gains.
Limitations
- Relies on a UMLS-derived knowledge graph and TransE/GAT entity embeddings, adding dependence on external expert knowledge.
- Gains over strongest baselines are often small at higher data fractions (e.g., 100% settings).
- Evaluation limited to chest radiograph datasets and frozen-backbone protocols.
Experimental setup
Datasets: MIMIC-CXR UMLS RSNA Pneumonia Object CXR SIIM Pneumothorax COVIDx CheXpert
Benchmarks: RSNA Pneumonia Object CXR SIIM Pneumothorax CheXpert COVIDx
Metrics: mAP Dice AUROC accuracy
Baselines: ConVIRT GLoRIA-CheXpert GLoRIA-MIMIC MGCA M-FLAG PRIOR DSVE VSE++ ImageNet Init Random Init
Builds on: GLoRIA MGCA ConVIRT TransE Sinkhorn-Knopp clustering
Closest papers by meaning
- KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · similarity 0.87)
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.86)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.86)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.86)
- Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.85)
- Randomized-MLP Regularization Improves Domain Adaptation and Interpretability in DINOv2 (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.