Research map / Vision-language and multimodal / Human and embodied modeling

A World Model of Radiologist Reading for Medical Image Representation Learning

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Medical vision-language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GazeWorld, a gaze-ordered JEPA-style world model that predicts latent representations of successive radiologist-fixated patches (plus spatial completion of unvisited patches) to refine medical image encoders, evaluated on chest X-ray diagnosis and GazeSearch scanpath probing.

Problem

Existing medical imaging methods use radiologist gaze only as static spatial priors or as a scanpath prediction target decoupled from diagnosis, leaving the temporal order of expert evidence gathering unused for representation learning.

Approach

Treats the image as a world and the radiologist's fixation sequence as a trajectory; an autoregressive JEPA-style predictor predicts the EMA-target latent of the next fixated patch from prior fixations, combined with a cross-attention spatial-completion branch predicting unvisited patch latents.

Key results

• Frozen GazeWorld features achieve best supervised AUROC in all nine CheXpert/RSNA/SIIM-ACR label settings (e.g., CheXpert 1% 78.37 vs EGMA 71.21).
• On GazeSearch, the best GazeWorld variant reaches ScanMatch 0.489 vs 0.419 for LogitGaze-Med (CheX) and SED 3.63 vs 4.68.
• Zero-shot accuracy improves over MedCLIP initialization by about 2 to 10 points across the three datasets when GazeWorld is applied (e.g., SIIM-ACR accuracy 56.89 to 66.59).

Surprising findings

• Raster-scan or random fixation ordering substantially degrades zero-shot AUROC, below chance on several datasets (e.g., RSNA 50.39 raster-scan).
• A generic scanpath decoder on frozen GazeWorld features outperforms purpose-built LogitGaze-Med despite no explicit gaze prediction training.

Limitations

• Limited to chest radiographs and a single eye-tracking corpus (MIMIC-EYE).
• Evaluation of localization is qualitative only (Grad-CAM), with no clinician-centered study.
• Baseline numbers for several methods are taken from EGMA or original papers, and backbones differ (ResNet-50 vs Swin), confounding comparisons.

Experimental setup

Datasets: MIMIC-EYE MIMIC-CXR GazeSearch CheXpert RSNA Pneumonia SIIM-ACR Pneumothorax

Benchmarks: CheXpert RSNA Pneumonia SIIM-ACR Pneumothorax CheXpert 5x200 GazeSearch

Metrics: AUROC accuracy F1 ScanMatch SED STDE MultiMatch

Baselines: BioViL MedKLIP MGCA GLoRIA PRIOR MedCLIP CheXWorld EGMA GazeFormer HAT GazeSearch LogitGaze-Med

Builds on: JEPA MedCLIP EGMA LogitGaze-Med CheXWorld

Closest papers by meaning

• Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis (AAAI 2024 · similarity 0.88)
• CheXWorld: Exploring Image World Modeling for Radiograph Representation Learning (CVPR 2025 · similarity 0.87)
• Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning (NeurIPS 2024 · similarity 0.87)
• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.86)
• MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis (AAAI 2026 · similarity 0.86)
• Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.85)