Research map / Vision-language and multimodal / Cross-modal learning
Improving Medical Multi-modal Contrastive Learning with Expert Annotations
Yogesh Kumar, Pekka Marttinen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add expert eye-gaze heatmap pairs with mixup, curriculum and priming (eCLIP) | CLIP ViT Base · CheXpert 5x200 zero-shot | macro F1: 0.540 -> 0.563 | Improved |
| add expert heatmaps naively to training set without mixup or curriculum (ablation) | CLIP ViT Base · CheXpert 5x200 zero-shot | macro F1: 0.540 -> 0.506 | Hurt |
| add expert heatmaps naively to GLoRIA training (ablation) | GLoRIA ResNet50 · CheXpert 5x200 zero-shot | macro F1: 0.478 -> 0.391 | Hurt |
| add DACL mixup during pretraining | CLIP Swin Tiny · CXR 14x100 zero-shot | macro F1: 0.169 -> 0.101 | Hurt |
| replace multi-headed-attention heatmap processor with random heatmaps (ablation) | eCLIP · CheXpert and CXR14 zero-shot | macro F1 | Hurt |
| use MHA heatmap processor instead of mask multiplication or CNN encoder (ablation) | eCLIP · CheXpert 5x200 and CXR 14x100 | macro F1 | Improved |
| add eCLIP expert examples on binary RSNA task | CLIP Swin Tiny · RSNA zero-shot | macro F1: 0.808 -> 0.818 | No clear effect |
| add expert annotations to CLIP on image-to-report retrieval | CLIP Swin Tiny · Open-I retrieval | Recall@10: 11.3 -> 13.4 | Improved |
| apply m3-mixup post-pretraining | CLIP Swin Tiny · Open-I retrieval | Recall@10: 11.3 -> 9.8 | Hurt |
| use eCLIP embeddings in RAG report generation | CLIP with Mistral 7B · Open-I report generation | BLEU-2: 0.172 -> 0.177 | Improved |
| add expert annotations with limited pretraining batches | CLIP Swin Tiny · MIMIC, CheXpert, CXR14 zero-shot | macro F1 | Improved |
Closest papers by meaning
- Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning (NeurIPS 2024 · similarity 0.86)
- MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning (CVPR 2024 · similarity 0.86)
- Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis (CVPR 2025 · similarity 0.86)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.85)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.85)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
