Research map / Vision-language and multimodal / Human and embodied modeling
Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills chest-related knowledge from a pre-trained 2D X-ray expert (CheXzero) into a 3D chest CT image encoder, using language-guided retrieval of paired X-rays, plus robust contrastive learning and entity-focused masking, trained on 12K CT-report pairs.
Problem
Building versatile 3D chest CT image understanding models is hard because large paired CT-report datasets are scarce, unlike chest X-ray image-report data.
Approach
Use reports as a bridge to retrieve, for each CT volume, the semantically closest MIMIC-CXR X-ray, then distill both pair-wise image embeddings and inter-sample relation matrices from the X-ray expert into the CT encoder; additionally correct false negatives in contrastive learning using report-consistency rules.
Key results
- Zero-shot classification on ChestCT-16K reaches 30.6 precision, 57.7 recall and 33.0 F1, versus 16.0 F1 for CLIP and 25.7 for DCL.
- Report generation (disease-diagnosis F1 averaged over six entities) reaches 37.3 on ChestCT-16K, above DCL (35.7) and BLIP (35.4).
- Reader study on 200 patients shows the model's nodule detection is comparable to a radiologist with over 10 years of experience.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BIUD instead of DCL for report generation | report generation · ChestCT-16K test | F1-score: 35.7 -> 37.3 | Helped |
| use BIUD instead of DCL for report generation | report generation · ChestCT-EXT | F1-score: 38.0 -> 40.6 | Helped |
| use BIUD instead of DCL for zero-shot diagnosis | zero-shot classification · ChestCT-16K test | F1-score: 25.7 -> 33.0 | Helped |
| use BIUD instead of DCL for pulmonary nodule detection | zero-shot classification · LIDC | F1-score: 56.0 -> 65.8 | Helped |
| add entity-focused masking (ablation) | BIUD baseline · report generation validation set | F1-score: 35.0 -> 36.7 | Helped |
| add robust contrastive learning (ablation) | BIUD baseline · report generation validation set | F1-score: 35.0 -> 36.9 | Helped |
| add entity-focused masking and robust contrastive learning (ablation) | BIUD baseline · report generation validation set | F1-score: 35.0 -> 37.5 | Helped |
| add knowledge distillation to entity-focused masking and robust contrastive learning (ablation) | BIUD · report generation validation set | F1-score: 37.5 -> 38.9 | Helped |
| add knowledge distillation to entity-focused masking and robust contrastive learning (ablation) | BIUD · report generation validation set | precision: 38.8 -> 37.7 | Hurt |
| add knowledge distillation to entity-focused masking and robust contrastive learning (ablation) | BIUD · report generation validation set | recall: 36.6 -> 40.8 | Helped |
| initialize fine-tuning with BIUD instead of ImageNet pre-training | multi-label classification · ChestCT-16K test | F1-score: 34.5 -> 39.0 | Helped |
Surprising findings
- Fine-tuning from ImageNet pre-training scored slightly higher than some report-generation results in the authors' comparison, which they attribute to language supervision being richer than one-hot labels.
- Adding knowledge distillation mainly raised recall (40.8) while precision slightly dropped (37.7) in the ablation.
Limitations
- Data come from a single private hospital, with external evaluation on only one other hospital's data.
- Zero-shot and report-generation performance is still modest in absolute terms (F1 around 37), so clinical application is far off per the authors.
- Only six entities are evaluated, and labels come from a modified CheXpert labeler rather than radiologist annotations.
Experimental setup
Datasets: ChestCT-16K MIMIC-CXR ChestCT-EXT LIDC
Benchmarks: ChestCT-16K ChestCT-EXT LIDC
Metrics: precision recall F1-score BLEU (excluded)
Baselines: R2Gen CMN RadFM BLIP DCL CLIP
Builds on: CheXzero CLIP BLIP relational knowledge distillation
Closest papers by meaning
- Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.92)
- AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.88)
- RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability (NeurIPS 2025 · similarity 0.87)
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.87)
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.87)
- Novel Class Discovery in Chest X-rays via Paired Images and Text (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.