Research map / Vision-language and multimodal / Cross-modal learning
Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EGMA, a medical vision-language pre-training framework that uses radiologists' synchronized eye-gaze and speech transcripts to guide fine-grained patch-sentence alignment and gaze-weighted cross-modality mapping.
Problem
Cross-modal alignment of medical images and diagnostic text is hard to learn from limited paired data, and implicit alignment learned from data generalizes poorly.
Approach
Uses radiologists' eye-gaze heatmaps aligned with transcribed sentences as explicit expert priors: a gaze-guided patch-sentence similarity loss (EGF) and a gaze-weighted cross-modality mapping loss (EGM) added on top of CLIP-style contrastive pre-training.
Key results
- Zero-shot retrieval on CheXpert 8x200: text-to-image P@1 80.00 vs 60.25 for ConVIRT (second best), a 19.75-point gain.
- Zero-shot classification on CheXpert 5x200 reaches 61.30 accuracy / 60.38 F1, above MedCLIP's 57.50 / 55.97.
- Using only 5% of gaze data (185 samples) improves over the baseline on all three zero-shot classification datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EGMA gaze-guided alignment | Medical multimodal pretraining · CheXpert 8x200 image-to-text retrieval | P@1: 38.75 -> 42.65 | Helped |
| Add EGMA gaze-guided alignment | Medical multimodal pretraining · CheXpert 8x200 text-to-image retrieval | P@1: 60.25 -> 80.00 | Helped |
| Use gaze-guided MLCE only (ablation) | EGMA · CheXpert 5x200 | accuracy: 57.50 -> 60.90 | Helped |
| Use gaze-guided MLCE only (ablation) | EGMA · RSNA | accuracy: 43.09 -> 47.06 | Helped |
| Use gaze-guided MLCE only (ablation) | EGMA · SIIM-ACR | accuracy: 58.40 -> 27.43 | Hurt |
| Add eye-gaze-guided fine-grained alignment (EGF) (ablation) | EGMA · CheXpert 5x200 | accuracy: 57.50 -> 60.30 | Helped |
| Add eye-gaze-guided fine-grained alignment (EGF) (ablation) | EGMA · SIIM-ACR | accuracy: 58.40 -> 63.54 | Helped |
| Add eye-gaze-guided cross-modal mapping (EGM) (ablation) | EGMA · RSNA | accuracy: 43.09 -> 54.68 | Helped |
| Add both EGF and EGM modules (ablation) | EGMA · RSNA | accuracy: 43.09 -> 76.97 | Helped |
| Train with 1% gaze data (ablation) | EGMA · CheXpert 5x200 | accuracy: 57.50 -> 58.93 | Helped |
| Train with 1% gaze data (ablation) | EGMA · RSNA | accuracy: 43.09 -> 40.38 | Hurt |
| Increase gaze-data share to 5% (ablation) | EGMA · SIIM-ACR | accuracy: 58.40 -> 59.20 | Helped |
Surprising findings
- The MLCE-only variant drops sharply on SIIM-ACR (27.43 accuracy vs 58.40 baseline), showing that gaze similarity loss alone can hurt generalization.
- The MGCA backbone with EGMA training lowers RSNA performance, which the authors attribute to shortcut features.
Limitations
- Evaluated only on classification and retrieval; no lesion localization or segmentation downstream tasks.
- Relies on MIMIC-EYE-style datasets that jointly record gaze, images and speech, which are scarce.
- Small gaze training set (3689 samples) and only chest X-ray domain.
Experimental setup
Datasets: MIMIC-EYE CheXpert MIMIC-CXR RSNA SIIM-ACR
Benchmarks: CheXpert RSNA SIIM-ACR CheXpert 5x200 CheXpert 8x200
Metrics: AUROC accuracy F1-score precision at top-k
Baselines: CLIP ConVIRT BioViL MedKLIP MGCA GLoRIA PRIOR MedCLIP
Builds on: CLIP GLoRIA MGCA
Closest papers by meaning
- Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis (AAAI 2024 · similarity 0.87)
- A World Model of Radiologist Reading for Medical Image Representation Learning (NeurIPS 2026 · similarity 0.87)
- ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models (NeurIPS 2025 · similarity 0.85)
- Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.85)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.85)
- MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.