Research map / Vision-language and multimodal / CLIP and zero-shot
RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RadZero, a chest X-ray vision-language model that uses cosine similarity-based cross-attention (VL-CABS) over patch features, multi-positive contrastive training on LLM-extracted finding-sentences, and zero-shot classification, grounding, and segmentation.
Problem
Existing medical VLMs underuse complex radiology reports (embedding whole reports or single words) and their attention-map explanations are unnormalized and inconsistent, limiting interpretable zero-shot chest X-ray alignment.
Approach
VL-CABS computes scaled cosine similarity between each sentence embedding and every image patch, softmax-attends patches for a global logit, and uses the raw patch similarities as a bounded, threshold-able pixel-level VL similarity map; combined with multi-positive NCE over all extracted finding-sentences per image.
Key results
- RadZero reaches average zero-shot classification AUC about 1.5 points above CARZero and the best PadChest score (0.841 vs 0.810 for CARZero).
- On CXD10 grounding, RadZero reaches mean pointing game accuracy 0.622 vs 0.543 for CARZero; on MS-CXR phrase grounding it reaches 0.844 vs 0.749.
- On SIIM zero-shot segmentation, RadZero reaches Dice 0.171 vs 0.100 for CARZero, and Pix-AUC 0.947 vs 0.856.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace scaled dot-product similarity with cosine similarity (ablation) | RadZero alignment · PadChest | AUC: 0.824 -> 0.830 | Helped |
| replace scaled dot-product similarity with cosine similarity (ablation) | RadZero alignment · ChestXDet10 | pointing game accuracy: 0.472 -> 0.483 | Helped |
| add two Transformer layers instead of a linear layer (ablation) | RadZero vision projection · ChestXDet10 | pointing game accuracy: 0.483 -> 0.539 | Helped |
| add two Transformer layers instead of a linear layer (ablation) | RadZero vision projection · SIIM | Dice: 0.078 -> 0.099 | Helped |
| use multi-positive contrastive pairs (ablation) | RadZero training · PadChest20 | AUC: 0.860 -> 0.879 | Helped |
| use multi-positive contrastive pairs (ablation) | RadZero training · SIIM | Dice: 0.099 -> 0.121 | Helped |
| increase input resolution from 224 to 518 (ablation) | RadZero · ChestXDet10 | pointing game accuracy: 0.537 → 0.622 | Helped |
| increase input resolution from 224 to 518 (ablation) | RadZero · SIIM | Dice: 0.121 → 0.171 | Helped |
| increase input resolution from 224 to 518 (ablation) | RadZero · PadChest20 | AUC: 0.879 -> 0.871 | Hurt |
| train on all view positions instead of frontal views only (ablation) | RadZero · PadChest | AUC: 0.831 -> 0.841 | Helped |
| use XrayDINOv2 at 518 instead of 224 resolution (ablation) | RadZero vision encoder · ChestXDet10 grounding | pointing game accuracy: 0.548 -> 0.622 | Helped |
| use two Transformer layers instead of a linear layer (ablation) | RadZero vision projection · SIIM | Dice: 0.100 -> 0.171 | Helped |
Surprising findings
- The 224px variant of RadZero outperforms the 518px model on several classification benchmarks, which the authors attribute to the encoder being pre-trained at 224px.
- CARZero's pre-softmax logits score lower Dice than its attention maps on SIIM (0.081 vs 0.100) despite threshold tuning.
Limitations
- Performance drops on some datasets (e.g., ChestXray14 and CXD10 classification) where CARZero is stronger.
- Validated only on chest X-ray; relies on a frozen pre-trained vision encoder that may limit domain adaptability.
- Abbreviations and rare expressions in reports yield weaker embedding alignment.
Experimental setup
Datasets: MIMIC-CXR MIMIC-CXR-JPG Open-I PadChest PadChest20 ChestXray14 CheXpert ChestXDet10 SIIM RSNA MS-CXR
Benchmarks: Open-I PadChest PadChest20 ChestXray14 CheXpert ChestXDet10 SIIM RSNA MS-CXR
Metrics: AUROC pointing game accuracy Dice Pixel-wise AUC
Baselines: GLoRIA BioViL-T MedKLIP KAD CARZero G2D MGCA LiT dino.txt
Builds on: LiT dino.txt UniCLIP multi-positive NCE CARZero GLoRIA
Closest papers by meaning
- CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification (CVPR 2024 · similarity 0.91)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.89)
- Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.87)
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.87)
- LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation (ICLR 2024 · similarity 0.87)
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.