Research map / Vision-language and multimodal / Human and embodied modeling
G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G2D, a medical vision-language pre-training framework that adds a parameter-free pseudo-segmentation pretext task, with masks derived from attention maps, to learn global and dense chest X-ray representations from image-report pairs.
Problem
Medical VLP methods align images with whole reports, which gives weak pixel-level supervision, so learned representations transfer poorly to dense prediction tasks such as segmentation and detection.
Approach
Build pseudo segmentation masks on the fly from the image encoder's multi-head attention maps (body-mask filtering, 85th-percentile thresholding, bilateral smoothing) and train an image decoder to predict them jointly with the vision-language contrastive loss, so the dense pretext task needs no extra trainable parameters or annotations.
Key results
- On SIIM segmentation, G2D with 1% of training data (Dice 65.6 with encoder-decoder) surpasses all peer methods trained on 100% of data.
- On CXR14 fine-tuned classification with 100% data, G2D reaches a mean AUC of 83.1 versus 82.5 for KAD (which uses extra annotations).
- Replacing the pseudo mask with a shuffled (semantically unpaired) mask drops SIIM Dice from 65.6 to 50.9, supporting that the mask semantics matter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pseudo segmentation instead of decoder-level reconstruction (ablation) | G2D · SIIM segmentation | Dice: 53.4$\pm$1.3 -> 65.6$\pm$1.7 | Helped |
| add a decoder-level pseudo-segmentation constraint (ablation) | G2D · SIIM segmentation | Dice: 49.2$\pm$1.5 -> 65.6$\pm$1.7 | Helped |
| set pseudo-mask threshold to the 85% percentile rather than 75% percentile (ablation) | G2D · SIIM segmentation | Dice: 63.0$\pm$2.1 -> 65.6$\pm$1.7 | Helped |
| set pseudo-mask threshold to the 85% percentile rather than median (ablation) | G2D · SIIM segmentation | Dice: 58.8$\pm$1.6 -> 65.6$\pm$1.7 | Helped |
| set pseudo-mask threshold to the 85% percentile rather than GMM (ablation) | G2D · SIIM segmentation | Dice: 59.2$\pm$1.5 -> 65.6$\pm$1.7 | Helped |
| use attention-map aggregation across heads (ablation) | G2D · SIIM segmentation | Dice: 62.1$\pm$2.2 -> 65.6$\pm$1.7 | Helped |
| use three attention heads instead of one or two (ablation) | G2D · SIIM segmentation | Dice: 63.4$\pm$2.0; 64.7$\pm$1.6 -> 65.6$\pm$1.7 | Helped |
| use four rather than three attention heads (ablation) | G2D · SIIM segmentation | Dice: 65.6$\pm$1.7 -> 65.3$\pm$1.6 | Hurt |
| use a 128-dimensional rather than 256-dimensional projector (ablation) | G2D · SIIM segmentation | Dice: 64.9$\pm$1.9 -> 65.6$\pm$1.7 | Helped |
| use a 256-dimensional rather than 128-dimensional projector (ablation) | G2D · RSNA object detection | mAP: 15.9$\pm$0.8 -> 16.1$\pm$1.1 | Helped |
| apply body-mask filtering (ablation) | G2D · SIIM segmentation | Dice: 63.4$\pm$1.5 -> 65.6$\pm$1.7 | Helped |
| apply edge smoothing (ablation) | G2D · SIIM segmentation | Dice: 64.1$\pm$1.2 -> 65.6$\pm$1.7 | Helped |
Surprising findings
- Pseudo masks built from attention maps without any annotation are semantically meaningful: visualizations show they capture regions tied to report words.
- Decoder-level pseudo-segmentation outperforms pixel reconstruction as a decoder pretext task on all three downstream tasks.
Limitations
- Pseudo masks are derived from weak language supervision and may not associate each pixel with specific text tokens, potentially capping performance.
- Under the reported frozen-encoder protocol, MedKLIP results were reimplemented by the authors, and comparisons depend on differing configurations across baselines.
Experimental setup
Datasets: MIMIC-CXR RSNA Pneumonia SIIM Object-CXR CheXpert CXR14 COVIDx
Benchmarks: SIIM RSNA Pneumonia Object-CXR CXR14 CheXpert COVIDx
Metrics: Dice mAP AUC F1 ACC recall IoU
Baselines: ConVIRT GLoRIA GLoRIA-MIMIC MGCA M-FLAG MedKLIP BioViL KAD CheXzero MRM ImageNet Init Random Init
Builds on: CLIP ConVIRT MedSAM U-Net ClinicalBERT
Closest papers by meaning
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.87)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.86)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.86)
- Training Like a Medical Resident: Context-Prior Learning Toward Universal Medical Image Segmentation (CVPR 2024 · similarity 0.86)
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.86)
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.