Research map / Vision-language and multimodal / Human and embodied modeling
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MedUnifier, a unified medical vision-language pre-training framework that adds a VQ-VAE-based text-grounded image generation objective to ITC, ITM and ITG losses, trained on MIMIC-CXR.
Problem
Existing medical vision-language pre-training emphasizes feature extraction and cross-modal understanding while neglecting text-grounded visual content generation, which limits multi-modal learning and fine-grained visual detail capture.
Approach
Integrates a hierarchical VQ-VAE image generator with latent adapters into a BLIP-2-style fusion encoder with learnable query embeddings, co-training image generation (TIG) end-to-end with contrastive, matching and text generation objectives using discrete visual representations.
Key results
- On MIMIC-CXR report generation, MedUnifier (w TIG) reaches BLEU-4 12.9 and ROUGE-L 34.0, above R2Gen (10.3 / 27.7).
- Fine-tuned RSNA AUROC at 1% data improves to 87.6 with TIG versus 86.3 without and 85.8 for MGCA.
- Image-to-text retrieval mAP@1 on MIMIC 5x200 rises to 60.7 with TIG versus 57.4 without and 52.4 for REFERS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TIG to MedUnifier (ablation) | MedUnifier · RSNA, SIIM, and COVIDx fine-tuning | AUC / accuracy: RSNA: 86.3, 87.1, 88.1 -> 87.6, 88.8, 91.7; SIIM: 87.6, 88.7, 92.3 -> 87.9, 92.5, 94.8; COVIDx: 75.3, 87.5, 92.8 -> 76.8 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR 5x200 ITR | mAP@1: 57.4 -> 60.7 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR 5x200 TIR | mAP@1: 59.6 -> 63.1 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR 5x200 TIR | mAP@5: 68.3 -> 70.8 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR 5x200 TIR | mAP@10: 63.4 -> 64.4 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC 5x200 zero-shot | accuracy: 44.8 -> 50.4 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · CheXpert 5x200 zero-shot | accuracy: 40.8 -> 43.5 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · RSNA zero-shot | accuracy: 85.0 -> 82.0 | Hurt |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR test set | BLEU-4: 11.3 -> 12.9 | Helped |
| Add TIG to MedUnifier (ablation) | MedUnifier · MIMIC-CXR test set | ROUGE-L: 29.6 -> 34.0 | Helped |
| Add ITM to ITC (ablation) | MedUnifier pre-training objectives · MIMIC-CXR 5x200 retrieval | mAP@1: 53.7 -> 55.2 | Helped |
| Add ITG to ITC and ITM (ablation) | MedUnifier pre-training objectives · MIMIC-CXR 5x200 retrieval | mAP@1: 55.2 -> 57.4 | Helped |
Surprising findings
- Excluding TIG slightly lowers text-to-image retrieval performance, which the authors attribute to over-reliance on ITG causing vision-language fusion imbalance.
- GLoRIA outperforms MedUnifier on zero-shot CheXpert 5x200 classification.
Limitations
- Generated synthetic images have FID 46.2, worse than RoentGen (42.4) and LLM-CXR (22.8) in the reported comparison.
- Evaluation is limited to chest X-ray datasets from radiology.
Experimental setup
Datasets: MIMIC-CXR v2.0.0 RSNA Pneumonia SIIM-ACR COVIDx CheXpert 5x200
Benchmarks: RSNA Pneumonia SIIM-ACR COVIDx MIMIC 5x200 CheXpert 5x200 MIMIC-CXR test set
Metrics: AUROC accuracy mAP@K BLEU METEOR ROUGE-L FID
Baselines: ConVIRT GLoRIA BioViL LoVT MGCA PRIOR MedCLIP REFERS CXR-CLIP R2Gen PTUnifier LLM-CXR
Builds on: BLIP-2 VQ-VAE BERT CLIP
Closest papers by meaning
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.88)
- HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation (ICML 2025 · similarity 0.88)
- MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants (NeurIPS 2025 · similarity 0.88)
- Modality-Specialized Synergizers for Interleaved Vision-Language Generalists (ICLR 2025 · similarity 0.88)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.87)
- Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.