AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Improving Medical Multi-modal Contrastive Learning with Expert Annotations

Yogesh Kumar, Pekka Marttinen

ECCV 2024 · Vision-language and multimodal › Cross-modal learning · approach: Unimodal-to-multimodal representation transfer · 29 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add expert eye-gaze heatmap pairs with mixup, curriculum and priming (eCLIP)CLIP ViT Base · CheXpert 5x200 zero-shotmacro F1: 0.540 -> 0.563Improved
add expert heatmaps naively to training set without mixup or curriculum (ablation)CLIP ViT Base · CheXpert 5x200 zero-shotmacro F1: 0.540 -> 0.506Hurt
add expert heatmaps naively to GLoRIA training (ablation)GLoRIA ResNet50 · CheXpert 5x200 zero-shotmacro F1: 0.478 -> 0.391Hurt
add DACL mixup during pretrainingCLIP Swin Tiny · CXR 14x100 zero-shotmacro F1: 0.169 -> 0.101Hurt
replace multi-headed-attention heatmap processor with random heatmaps (ablation)eCLIP · CheXpert and CXR14 zero-shotmacro F1Hurt
use MHA heatmap processor instead of mask multiplication or CNN encoder (ablation)eCLIP · CheXpert 5x200 and CXR 14x100macro F1Improved
add eCLIP expert examples on binary RSNA taskCLIP Swin Tiny · RSNA zero-shotmacro F1: 0.808 -> 0.818No clear effect
add expert annotations to CLIP on image-to-report retrievalCLIP Swin Tiny · Open-I retrievalRecall@10: 11.3 -> 13.4Improved
apply m3-mixup post-pretrainingCLIP Swin Tiny · Open-I retrievalRecall@10: 11.3 -> 9.8Hurt
use eCLIP embeddings in RAG report generationCLIP with Mistral 7B · Open-I report generationBLEU-2: 0.172 -> 0.177Improved
add expert annotations with limited pretraining batchesCLIP Swin Tiny · MIMIC, CheXpert, CXR14 zero-shotmacro F1Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy