Research map / Vision-language and multimodal / Multimodal LLMs
QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QoQ-Med-7B/32B, an open multimodal clinical reasoning model (images, ECG time series, text) trained with DRPO, a critic-free RL method that hierarchically rescales GRPO advantages by domain and within-domain reward clusters.
Problem
Existing medical MLLMs are vision-centric, do not integrate time-series signals such as ECG with imaging, and do not expose reasoning or localize evidence; heterogeneous clinical data distributions also cause critic-free RL (GRPO) to be dominated by abundant domains.
Approach
DRPO extends GRPO with hierarchical temperature scaling of normalized advantages: inter-domain scaling by domain size and mean reward, plus intra-domain scaling by K-means clusters of per-question reward vectors, so rare and hard domains and clusters get more weight; the model also outputs chain-of-thought, bounding boxes, and diagnosis, with IoU-based reward.
Key results
- DRPO reaches overall macro-F1 of 0.295 vs 0.193 for GRPO on the eight-vision-domain CLIMB evaluation (averaged over four seeds), with the paper claiming a 43% average F1 gain.
- QoQ-Med-7B attains overall accuracy 0.686 on CLIMB vision domains, above GPT-4o (0.385) and o4-mini (0.411) in the appendix comparison.
- On MIMIC-IV 48-hour mortality, DRPO-Full reaches accuracy 0.642 and F1 0.597 versus 0.551 and 0.354 for GRPO-Full.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DRPO instead of GRPO | Qwen2.5-VL-7B · Overall, 8 medical imaging domains | F1: .193 -> .295 | Helped |
| Add domain-wise scaling to GRPO (ablation) | DRPO · Overall, 8 medical imaging domains | F1: 22.8% improvement | Helped |
| Add within-domain clustering to domain-wise scaling (ablation) | DRPO · Overall, 8 medical imaging domains | F1: 19.4% | Helped |
| Remove KL-aware regularization (ablation) | DRPO · Overall, 8 medical imaging domains | F1: .295 -> .283 | Hurt |
| Use all three input modalities instead of text only (ablation) | DRPO on MIMIC-IV · LOS | F1: 0.195 -> 0.283 | Helped |
| Use all three input modalities instead of ECG and text (ablation) | DRPO on MIMIC-IV · 48-IHM | F1: 0.528 -> 0.597 | Helped |
| Use all three input modalities instead of vision and text (ablation) | DRPO on MIMIC-IV · 48-IHM | F1: 0.586 -> 0.597 | Helped |
| Set cluster count to 1 instead of 10 (ablation) | DRPO · Overall, 8 medical imaging domains | F1: .286 -> .259 | Hurt |
| Increase accuracy:IoU reward weights from 0.2:0.6 to 0.6:0.2 (ablation) | DRPO · Overall, 8 medical imaging domains | F1: .260 -> .286 | Helped |
Surprising findings
- Auxiliary formatting rewards saturate early and have effectively no impact later in training due to normalization.
- Among MIMIC-IV inputs, ECG contributes less to accuracy and F1 than vision and text, and the text-only ablation performs worst.
Limitations
- Reasoning is not supervised, so sample efficiency is limited.
- Performance on understudied modalities like ECG and ultrasound remains weak, and F1 remains low on many domains.
- Not suitable for clinical deployment without real-world testing.
Experimental setup
Datasets: CLIMB PTB-XL CheXpert MIMIC-CXR MIMIC-IV ISIC-2020 HAM10000 BUSI APTOS 2019
Benchmarks: CLIMB MIMIC-IV
Metrics: accuracy macro-F1 IoU
Baselines: SFT PPO ReMax Reinforce++ RLOO GRPO LLaVA-Med Med-R1 GPT-4o o4-mini
Builds on: GRPO PPO Qwen2.5-VL ECG-JEPA CLIMB
Closest papers by meaning
- MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images (CVPR 2026 · similarity 0.89)
- MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis (CVPR 2026 · similarity 0.88)
- MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.88)
- Token-Sparse Medical Multimodal Reasoning via Dual-Stream Reinforcement Learning (ICML 2026 · similarity 0.88)
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.88)
- MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.