Research map / Vision-language and multimodal / Human and embodied modeling
Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes McGIP, a plug-and-play module that uses radiologists' eye-tracking gaze similarity to create extra positive pairs in contrastive pre-training (BYOL, MoCo, SimSiam) for medical image classification on INbreast mammography and Tufts dental X-rays.
Problem
Contrastive pre-training for medical images relies on augmentation-based positive pairs; radiology reports are hard to obtain and classification labels are coarse and can collapse representations, so an alternative source of semantic positive pairs is needed.
Approach
Treat images that elicit similar radiologist gaze patterns as positive pairs, using three similarity schemes (multi-match on gaze sequences, Hu-moment-based gaze moment on heatmaps, and dHash on heatmaps for structured images), with a weighted contrastive loss over thresholded affinity.
Key results
- On INbreast with ResNet50, BYOL+McGIP reached 97.07 M-AUC and 87.57 ACC versus 93.84 M-AUC and 85.95 ACC for BYOL.
- On Tufts with ResNet18, BYOL+McGIP achieved 62.91 AUC and 65.00 ACC versus 60.61 AUC and 63.00 ACC for BYOL.
- Averaged over three backbones, MoCo+McGIP improved ACC from 83.42% to 85.41% on INbreast relative to MoCo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add McGIP to MoCo | MoCo with ResNet50 · INbreast | ACC: 81.62±1.38 -> 85.68±1.38 | Helped |
| Add McGIP to BYOL | BYOL with ResNet50 · INbreast | M-AUC: 93.84±1.72 -> 97.07±0.75 | Helped |
| Add McGIP to BYOL | BYOL with ResNet50 · Tufts | AUC: 52.12 -> 61.35 | Helped |
| Use gaze moment instead of multi-match gaze-sequence similarity (ablation) | BYOL+McGIP with ResNet50 · INbreast fine-tuning | M-AUC: 97.07±0.75 -> 95.75±1.10 | Hurt |
| Use gaze moment instead of multi-match gaze-sequence similarity (ablation) | BYOL+McGIP with ResNet50 · INbreast linear probing | M-AUC: 78.31±2.89 -> 82.20±2.18 | Helped |
| Use gaze-based positive pairs instead of diagnosis-label positive pairs | BYOL with ResNet50 · INbreast | AUC: 88.96±1.61 -> 93.80±0.79 | Helped |
| Use gaze-based positive pairs instead of diagnosis-label positive pairs | BYOL with ResNet50 · Tufts | ACC: 58.00 -> 67.5 | Helped |
| Use gaze-based positive pairs instead of diagnosis-label positive pairs | BYOL with ResNet101 · INbreast | AUC: 90.60±2.97 -> 90.09±3.08 | Hurt |
| Increase gaze-similarity threshold to 0.8 or 0.9 (ablation) | McGIP with BYOL and ResNet50 · INbreast | linear-probing performance | Hurt |
| Lower gaze-similarity threshold below 0.6 (ablation) | McGIP with BYOL and ResNet50 · INbreast | linear-probing performance | Hurt |
| Increase pre-training epochs beyond 300 (ablation) | McGIP with BYOL and ResNet50 · INbreast | linear-probing performance | No clear effect |
Surprising findings
- Gaze-based positive pairs outperformed ground-truth class-label supervised contrastive learning (BYOL+Sup) on INbreast in most settings, and on Tufts by a larger margin.
- Under linear probing, gaze moment and multi-match performed differently across metrics, with gaze moment better on M-AUC (82.20 vs 78.31) but multi-match better on ACC.
Limitations
- Experiments are limited to two datasets; the gaze data for INbreast comes from a single radiologist with ten years of experience.
- Gaze similarity thresholds and confidence p are manually set hyperparameters.
- Reported differences are often within overlapping standard deviations, and ResNet101 AUC on INbreast was slightly lower than baselines.
Experimental setup
Datasets: INbreast Tufts dental dataset
Benchmarks: INbreast Tufts dental dataset
Metrics: ACC AUC M-AUC
Baselines: From-scratch ImageNet pre-training MoCo BYOL SimSiam BYOL+Sup
Builds on: MoCo BYOL SimSiam multi-match Hu moments
Closest papers by meaning
- A World Model of Radiologist Reading for Medical Image Representation Learning (NeurIPS 2026 · similarity 0.88)
- Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning (NeurIPS 2024 · similarity 0.87)
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation (CVPR 2025 · similarity 0.85)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.84)
- MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis (AAAI 2026 · similarity 0.84)
- AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.