Research map / Vision-language and multimodal / Multimodal LLMs

PatientVLM Meets DocVLM: Pre-Consultation Dialogue Between Vision-Language Models for Efficient Diagnosis

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Simulates multi-turn doctor-patient dialogues between two VLMs (DocVLM asks questions, PatientVLM answers from the ground-truth diagnosis) and fine-tunes the DocVLM on the resulting image-dialogue-diagnosis triplets to improve medical image diagnosis on MedMNIST tasks.

Problem

Image-only medical diagnosis ignores patient-reported symptoms, and collecting real doctor-patient dialogues is impractical due to ethics, privacy and time constraints.

Approach

Two distinct VLMs play doctor and patient in a simulated, visually grounded consultation; the PatientVLM is conditioned on the ground-truth diagnosis without revealing it, and the resulting synthetic dialogues are used to LoRA-fine-tune the DocVLM for diagnosis prediction conditioned on image and dialogue history.

Key results

• PCDF-enhanced InternVL3-2B reaches 89.6% accuracy and 73.7 F1 on DermaMNIST, a +37.2 F1 gain over image-only SFT.
• PCDF-enhanced MedGemma3-4B improves F1 on RetinaMNIST from 71.2 to 81.3 over image-only SFT.
• Clinical validation on 210 cases (1,680 exchanges): 96.9% of exchanges rated clinically relevant by licensed experts, with no diagnosis leakage reported.

Surprising findings

• Zero-shot PCDF dialogues without dialogue-conditioned finetuning already improve F1 over zero-shot for MedGemma and Qwen2.5-VL, and in some cases beat chain-of-thought prompting.

Limitations

• Clinical validation was small-scale (210 cases) with limited medical professionals and no diverse patient populations.
• Some generated follow-up questions are overly technical for lay patients.
• The system supports only English.

Experimental setup

Datasets: DermaMNIST PneumoniaMNIST RetinaMNIST PathMNIST

Benchmarks: DermaMNIST PneumoniaMNIST RetinaMNIST PathMNIST

Metrics: accuracy F1

Baselines: ResNet50 DenseNet201 CLIP MedCLIP PMC-CLIP BioMedCLIP InternVL3-2B Qwen2.5-VL-7B Gemma3-4B MedGemma3-4B

Builds on: MedMNIST v2 LoRA mPLUG-Owl3

Closest papers by meaning

• CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework (ICLR 2026 · similarity 0.88)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.87)
• DrVD-Bench: Do Vision-Language Models Reason Like Human Doctors in Medical Image Diagnosis? (NeurIPS 2025 · similarity 0.87)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.87)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.87)