Research map / Vision-language and multimodal / Multimodal LLMs
Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MedRegInstruct, a region-centric medical instruction dataset (~550K region-text pairs plus ~240K grounded reports), and trains MedRegA, a bilingual region-aware medical MLLM on InternVL 1.2 that handles VQA, report generation, classification, and region identification/detection/grounded report generation.
Problem
Medical generalist MLLMs are region-agnostic: they treat the whole scan holistically and cannot identify, localize, or ground descriptions to specific anatomical regions or lesions, which limits interpretability and clinical usability.
Approach
Formulate three region-centric tasks (region-to-text identification, text-to-region detection, grounded report generation) with bounding-box tokens, build a semi-automatically labeled bilingual dataset for them, and train a single medical MLLM on it together with standard medical VQA, report, and classification corpora; add a regional chain-of-thought inference step that first detects regions then
Key results
- On Text-to-Region detection over single-object single-region, MedRegA reached 45.11% region accuracy and 42.70 IoU, versus InternVL's 7.24 and 16.61.
- On chest X-ray grounded report generation, MedRegA achieved BLEU-1 33.18 and region accuracy 76.59, versus MedDr's BLEU-1 27.43 and no region output.
- MedRegA reports 40.46 BLEU-1 on English report generation versus 34.49 for MedDr, and 40.76 vs 11.99 BLEU-1 on Chinese report generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add region-centric training data and region-aware capabilities | MedRegA · English report generation | BLEU-1: 5.97% | Helped |
| Add region-centric training data and region-aware capabilities | MedRegA · Chinese report generation | BLEU-1: 28.77% | Helped |
| Add Regional CoT using detected regions to guide generation | MedRegA · VinDr-SpineXR | F1 score: 34.95% and 31.59% | Helped |
| Use region-aware MedRegA instead of MedDr | Medical image single-label classification · Medical image classification | F1 Score: 16.99 to 47.97 | Helped |
| Use region-aware MedRegA instead of MedDr | Medical image multi-label classification · Medical image classification | F1 Score: 9.38 to 13.32 | Helped |
| Use region-aware MedRegA instead of MedDr | English medical VQA · English medical VQA | BLEU-1: 65.72 to 67.65 | Helped |
| Use region-aware MedRegA instead of MedDr | Chinese medical VQA · Chinese medical VQA | BLEU-1: 37.76 to 60.89 | Helped |
| Use MedRegA instead of MedDr | PathVQA · PathVQA | BLEU-1: 61.40 to 59.52 | Hurt |
Surprising findings
- Region-to-text identification scores for baselines such as InternVL are near zero (BLEU-1 0.13), suggesting general MLLMs cannot interpret bounding-box prompts in medical scans.
- Recall exceeds precision in multi-region detection, indicating the model tends to output fewer boxes than needed.
Limitations
- Automatic bounding box labels for in-house data had only 72% accuracy versus human annotation, and boxes were often slightly larger than targets.
- Training data is largely Chinese in-house data from one hospital, and the Ethics Statement says only constructed data derived from public datasets will be released.
- Baselines were evaluated with reproduced checkpoints, and InternVL was used as a stand-in baseline for region tasks since other medical MLLMs cannot output coordinates.
Experimental setup
Datasets: MedRegInstruct SA-Med2D-20M Chest ImaGenome MIMIC-CXR IU-Xray SLAKE VQA-RAD PathVQA PMC-VQA PMC-OA QUILT VinDr-CXR
Benchmarks: SLAKE VQA-RAD PathVQA PMC-VQA MIMIC-CXR IU-Xray in-house Chinese report test set MURA ISIC2018 VinDr-SpineXR VinDr-CXR MedRegInstruct region tasks
Metrics: BLEU-1 BLEU-4 F1 BertScore METEOR ROUGE-L accuracy recall IoU region F1
Baselines: Med-Flamingo LLaVA-Med RadFM MedDr BiomedGPT InternVL
Builds on: InternVL 1.2 SA-Med2D-20M Chest ImaGenome MedDr RadFM
Closest papers by meaning
- MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.91)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.91)
- MIMO: A Medical Vision Language Model with Visual Referring Multimodal Input and Pixel Grounding Multimodal Output (CVPR 2025 · similarity 0.90)
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.90)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.90)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.