Research map / Vision-language and multimodal / CLIP and zero-shot
Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes fVLM, a fine-grained CLIP-style pre-training method that aligns anatomy-level CT regions with per-anatomy report descriptions and corrects false negatives via a dual reduction and co-teaching scheme, trained on the 69K-patient MedVL-CT69K dataset.
Problem
Global image-report contrastive pre-training for CT misaligns local anatomical regions with report sentences, and fine-grained anatomy-level alignment suffers from many false negatives among normal and similarly diseased samples.
Approach
Match anatomy segmentation masks to per-anatomy report descriptions and run contrastive learning per anatomy; identify normal anatomies from the impression section and correct contrastive targets using similarity predictions from a co-taught second model.
Key results
- Zero-shot average AUC of 81.3% across 54 abnormality tasks on MedVL-CT69K, versus CLIP 68.4% (+12.9) and best supervised baseline 76.7% (+8.0 over supervised baseline comparison reported as 8.0).
- On CT-RATE internal validation zero-shot AUC 77.8 vs CT-CLIP 70.4 (+7.4); on Rad-ChestCT external AUC 68.0 vs 63.2 (+4.8).
- Frozen-encoder report generation ACC 61.5 vs CLIP 57.3 and Merlin 58.8 on MedVL-CT69K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fVLM instead of CLIP | Zero-shot abnormality detection · MedVL-CT69K | AUC: +12.9 points | Helped |
| Use fVLM instead of Merlin | Zero-shot abnormality detection · MedVL-CT69K | AUC: +9.4 points | Helped |
| Use fVLM instead of CT-CLIP | Zero-shot abnormality detection · CT-RATE internal evaluation | AUC: 7.4% | Helped |
| Use fVLM instead of CT-CLIP | Zero-shot abnormality detection · Rad-ChestCT external evaluation | AUC: 4.8% | Helped |
| Use fVLM instead of CLIP | Frozen-encoder report generation · MedVL-CT69K | ACC: 4.2 points | Helped |
| Use fVLM instead of CLIP | Frozen-encoder report generation · MedVL-CT69K | GREEN: 3.8 points | Helped |
| Use fVLM instead of CLIP | Fine-tuned report generation · MedVL-CT69K | ACC: 2.5-point improvement | Helped |
| Use fVLM instead of CLIP | Fine-tuned report generation · MedVL-CT69K | GREEN: 2.6-point improvement | Helped |
| Add fine-grained alignment and normal false-negative correction (ablation) | fVLM · MedVL-CT69K validation | AUC: 7.8 points | Helped |
| Add fine-grained alignment and normal false-negative correction (ablation) | fVLM · MedVL-CT69K validation | ACC: 6.0 points | Helped |
| Use co-teaching rather than the training model itself for label correction (ablation) | fVLM contrastive pre-training · MedVL-CT69K | diagnostic performance | Helped |
| Use co-teaching rather than a momentum model for label correction (ablation) | fVLM contrastive pre-training · MedVL-CT69K | diagnostic performance | Helped |
Surprising findings
- Implicit local alignment methods LoVT and MGCA gave only marginal gains over CLIP on CT, suggesting implicit local alignment is insufficient for 3D CT.
- Using the training model itself to correct contrastive labels degrades performance due to error accumulation, while the momentum model gives smaller gains than co-teaching.
Limitations
- Requires anatomy segmentation and LLM-based report decomposition, adding resource and time cost.
- Directly applying the fine-grained model to whole-report generation suffers granularity mismatch.
- Still a notable performance gap compared with professional radiologists in the reader study.
Experimental setup
Datasets: MedVL-CT69K CT-RATE Rad-ChestCT
Benchmarks: MedVL-CT69K CT-RATE Rad-ChestCT
Metrics: AUC ACC Spec Sens F1 Prec BLEU-1 BLEU-4 METEOR ROUGE-L
Baselines: CLIP LoVT MGCA Imitate ASG CT-GLIP BIUD Merlin CT-CLIP CT-VocabFine CT-LiPro
Builds on: CLIP MAE BERT TotalSegmentator
Closest papers by meaning
- Bootstrapping Chest CT Image Understanding by Distilling Knowledge from X-ray Expert Models (CVPR 2024 · similarity 0.92)
- Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.90)
- Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging (NeurIPS 2025 · similarity 0.88)
- VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · similarity 0.88)
- Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.87)
- AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.