Research map / Vision-language and multimodal / Human and embodied modeling

HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation

ICML 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HealthGPT, a unified medical vision-language model for comprehension and generation built by adapting Phi-3-mini/Phi-4 with heterogeneous LoRA (H-LoRA), hierarchical visual features, and a three-stage training pipeline on the VL-Health dataset.

Problem

Unifying medical visual comprehension and generation in one LVLM is underexplored, and mixed autoregressive training causes conflicts that degrade comprehension and generation performance under limited medical data.

Approach

Decouple comprehension and generation via task-gated heterogeneous LoRA plugins (hard routing per task type) with MoE-style LoRA experts merged by concatenation for efficiency, feed task-specific shallow or deep CLIP features, and train in three stages (alignment, shared embedding/head fine-tuning, instruction tuning with only H-LoRA trained).

Key results

• HealthGPT-L14 (14B) averages 66.4 on the medical comprehension table versus 54.7 for Llama-3.2 (11B) and 33.8 for Unified-IO 2.
• On OmniMedVQA, HealthGPT-L14 reaches 74.4 average versus 63.2 for Llama-3.2 and 50.0 for HuatuoGPT-Vision.
• H-LoRA with four experts trains at 1.00x time versus 1.49x for MoELoRA while improving OmniMedVQA comprehension from 64.90 to 68.50.

Measured effects reported in the paper

Surprising findings

• Mixed training of comprehension and generation data degrades comprehension strongly (e.g., VQA-RAD close 56.6 vs 72.5 with three-stage training), showing the task conflict the method targets.
• MoELoRA with four experts did not outperform plain LoRA on several comprehension tasks despite added capacity.

Limitations

• Loss spikes observed during training with larger parameters and learning rates.
• Generation evaluation covers limited tasks (modality conversion, super-resolution, reconstruction), and the report-to-CXR case is qualitative.
• Unified model still trails medical-specific comprehension models on some tasks such as MMMU-Med (HealthGPT-M3 68.5 vs Llama-3.2 63.2 on some; Med-specific lower), and human evaluation covers only 1,000 questions with 5 clinicians.

Experimental setup

Datasets: VL-Health PubMedVision LLaVA-Med PathVQA MIMIC-CXR-VQA SLAKE VQA-RAD LLaVA-1.5 LLaVA-558k IXI SynthRAD2023 MIMIC-CXR

Benchmarks: VQA-RAD SLAKE PathVQA MMMU-Med OmniMedVQA IXI super-resolution SynthRAD2023 modality conversion

Metrics: accuracy SSIM PSNR MSE LPIPS training time

Baselines: Med-Flamingo LLaVA-Med HuatuoGPT-Vision BLIP-2 LLaVA-v1.5 InstructBLIP Yi-VL InternVL2 Llama-3.2 Show-o Unified-IO 2 Janus

Builds on: LoRA MoELoRA VQGAN LLaVA

Closest papers by meaning

• MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.90)
• GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and a Comprehensive Multimodal Dataset Towards General Medical AI (AAAI 2026 · similarity 0.89)
• Enhancing Medical Large Vision-Language Models via Alignment Distillation (AAAI 2026 · similarity 0.88)
• Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE (NeurIPS 2024 · similarity 0.88)
• MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.88)
• VILA-M3: Enhancing Vision-Language Models with Medical Expert Knowledge (CVPR 2025 · similarity 0.88)