Research map / Vision-language and multimodal / CLIP and zero-shot
CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CARZero, a radiology vision-language model that uses cross-attention to produce a similarity representation (SimR) projected into an image-text similarity matrix for InfoNCE training, plus LLM-based prompt alignment, reaching SOTA zero-shot CXR classification AUC on five test sets.
Problem
Zero-shot chest radiograph classification relies on cosine similarity for image-text alignment, which may not capture the complex relationships between medical images and radiology reports; prompt design also strongly affects zero-shot performance.
Approach
Use cross-attention between global/local image features and sentence/word text features to generate a high-dimensional Similarity Representation (SimR), linearly project it to a similarity matrix for InfoNCE, and use an LLM (Spark) to insert standardized diagnostic prompt templates into training reports so training and inference prompts match.
Key results
- CARZero reaches AUC 0.810 on PadChest, 0.837 on PadChest20 and 0.811 on ChestXray14, outperforming the best prior method KAD (0.750, 0.735, 0.789).
- On ChestXray14, CARZero's zero-shot AUC (0.811) exceeds the 1%-data fine-tuned MRM result (0.794).
- Ablation on ChestXray14 shows cross-attention alignment and prompt alignment each improve AUC over the CLIP-style baseline (0.764 to 0.795 and 0.764 to 0.781), with both combined reaching 0.810.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CARZero instead of KAD | zero-shot classification · Open-I | AUC: 0.807 -> 0.838 | Helped |
| apply CARZero instead of KAD | zero-shot classification · PadChest | AUC: 0.750 -> 0.810 | Helped |
| apply CARZero instead of KAD | zero-shot classification · PadChest20 | AUC: 0.735 -> 0.837 | Helped |
| apply CARZero instead of KAD | zero-shot classification · ChestXray14 | AUC: 0.789 -> 0.811 | Helped |
| apply CARZero instead of KAD | zero-shot classification · CheXpert | AUC: 0.905 -> 0.923 | Helped |
| apply CARZero instead of KAD | zero-shot classification · ChestXDet10 | AUC: 0.735 -> 0.796 | Helped |
| apply CARZero instead of MedKLIP | zero-shot grounding · ChestXDet10 | pointing game (Mean): 0.481 -> 0.543 | Helped |
| add prompt alignment using P1 (ablation) | CLIP baseline · ChestXray14 | AUC: 0.764 -> 0.795 | Helped |
| add cross-attention alignment using P1 (ablation) | CLIP baseline · ChestXray14 | AUC: 0.764 -> 0.781 | Helped |
| use prompt alignment together with cross-attention alignment (ablation) | CLIP baseline · ChestXray14 | AUC: 0.764 -> 0.810 | Helped |
| replace P1 with empirically optimized P2 when using cross-attention alignment (ablation) | CLIP baseline · ChestXray14 | AUC: 0.781 -> 0.801 | Helped |
| use local rather than global features for cross-attention key and value (ablation) | CARZero · ChestXray14 | AUC: 0.799 -> 0.810 | Helped |
Surprising findings
- Replacing the cross-attention SimR with cosine similarity on projected features (cos(xy,yx)) gives comparable AUC (0.804 vs 0.810) but much lower MCC and F1 in the other variant, indicating cosine-based alignment is weaker.
Limitations
- Evaluated only on chest radiograph zero-shot classification; generalization to natural images and fine-tuning is untested.
- Relies on an LLM (Spark) to generate prompt templates, which is a proprietary, non-reproducible component.
- Uses translated PadChest reports, adding a translation step that may affect text quality.
Experimental setup
Datasets: MIMIC-CXR PadChest
Benchmarks: Open-I PadChest PadChest20 ChestXray14 CheXpert ChestXDet10
Metrics: AUC MCC F1 ACC pointing game
Baselines: MedCLIP BiomedCLIP GLoRIA BioViL CheXzero MedKLIP KAD MGCA MRM
Builds on: CLIP ConVIRT GLoRIA M3AE BioBERT
Closest papers by meaning
- RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability (NeurIPS 2025 · similarity 0.91)
- Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · similarity 0.86)
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.85)
- Novel Class Discovery in Chest X-rays via Paired Images and Text (AAAI 2024 · similarity 0.85)
- Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis (CVPR 2025 · similarity 0.85)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.