Research map / Vision-language and multimodal / Human and embodied modeling

Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation

NeurIPS 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Medical vision-language models · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ViMed-PET, a Vietnamese PET/CT image-report dataset of 2,757 whole-body volumes with augmentation and an expert-validated lung cancer clinical test set, and benchmarks CT-ViT and Cosmos Tokenizer encoders with Mistral-7B and LLaMA-2-7B on report generation and VQA.

Problem

Medical vision-language models cover few imaging modalities (PET/CT largely absent) and few low-resource languages (Vietnamese), and standard NLP metrics poorly reflect clinical accuracy of generated PET/CT reports.

Approach

A Vietnamese PET/CT image-report dataset with body-region partitioning, GPT-4o-based report paraphrase and study-comparison augmentation, and a clinical F1 metric over lesion type, position, and FDG uptake; VLMs are built by aligning 3D encoders (CT-ViT, adapted Cosmos Tokenizer) to LLMs in three stages with LoRA.

Key results

• Fine-tuned CT-ViT + Mistral-7B reaches BLEU-4 58.07 and F1-TPF 22.65 on report generation, versus 31.12 BLEU-4 for few-shot GPT-4o (where F1-TPF is 7.87).
• On VQA, CT-ViT + Mistral-7B achieves BLEU-4 31.14 and BERT 82.50, about 10.3x GPT-4o's BLEU-4 (3.01).
• Baselines LLaVA-Med, M3D and RadFM score near-zero BLEU-4 (0.01-0.06) on Vietnamese PET/CT report generation before fine-tuning.

Measured effects reported in the paper

Surprising findings

• Adding the study comparison augmentation gives no consistent gain on NLP metrics for some configurations, and the Cosmos Tokenizer's clinical F1 scores fall in the O-G-C setting for Mistral-7B.
• Cosmos Tokenizer, which is general-purpose, matches CT-ViT on NLP metrics, but CT-ViT is better on clinical F1 in every configuration.

Limitations

• Clinical F1 scores hover around 50% or less, showing models still underperform on clinically critical criteria such as lung cancer diagnosis.
• The clinical test set covers only 80 lung cancer patients, and the work uses only PET data for benchmarking despite having CT volumes.
• Reports from a single hospital and standardized templates may limit output diversity and generalization.

Experimental setup

Datasets: ViMed-PET CT-ViT pretraining chest CT data (25,701 volumes)

Benchmarks: ViMed-PET test set (1,725 pairs) ViMed-PET medical test set (80 lung cancer patients, 398 lesions)

Metrics: BLEU-4 ROUGE-1 ROUGE-L BERTScore F1-T F1-TP F1-TF F1-TPF

Baselines: LLaVA-Med M3D RadFM GPT-4o

Builds on: LLaVA-Med M3D CLIP LoRA CT-ViT

Closest papers by meaning

• GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.91)
• Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding (ICLR 2025 · similarity 0.90)
• VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · similarity 0.89)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.89)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.89)
• Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.89)