Research map / Vision-language and multimodal / Multimodal LLMs

OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies SFT data recipes for multimodal medical reasoning, distilling 8M reasoning traces from GPT-4o and DeepSeek-R1 into Qwen2.5-VL-7B to produce OctoMed-7B, which reports state-of-the-art open-source results on medical benchmarks.

Problem

Medical multimodal reasoning models are sensitive to how SFT training data is curated (question sources, prompt format, filtering, number of reasoning traces, teacher choice), yet these design choices are underexplored; the paper asks which data recipe most improves medical reasoning robustness.

Approach

A data-centric recipe for medical multimodal SFT: mix text-only, multimodal reasoning, and classification questions; distill CoT traces from reasoning and instruction-tuned teachers with rejection sampling; keep up to 16 traces per question; scale to 8M examples. The mixture makes the model self-calibrate reasoning length to task difficulty.

Key results

• OctoMed-7B reaches 67.83 overall on text-only benchmarks and 67.29 on multimodal classification, exceeding the open-source 7–9B baselines in the table and MedGemma-27B overall.
• Increasing rejection samples per MedQA question from 1 to 16 raises peak MedQA test accuracy from 75.16 to 85.01.
• CoT prompting improves multimodal reasoning from 23.08 to 38.15 overall, while direct prompting is better on classification (65.46 vs 63.33).

Surprising findings

• OctoMed-7B's multimodal classification overall (67.29) surpasses GPT-4o (53.96), the teacher used for multimodal data.
• Fine-tuning reasoning-post-trained models (InternVL3.5-8B) gave smaller text and reasoning gains than fine-tuning instruction-tuned models.

Limitations

• Evaluation uses GPT-4.1-mini as judge for some checkpoints, and some baseline models needed custom prompts.
• Question-filtering gains vanish at convergence, and the study focuses on SFT without RL.
• Some comparisons use reduced training subsets (e.g., 100k or 30k examples) rather than the full mixture.

Experimental setup

Datasets: MedQA HeadQA MedMCQA PMC-VQA MMMU-PRO MedXpertQA NEJM Image Challenge Brain Tumor CoronaHack Aptos MESSIDOR-2 BCSS

Benchmarks: MedQA HeadQA MedMCQA MedXpertQA MMLU-PRO (health) MMMU-PRO (medical) NEJM Image Challenge PMC-VQA Brain Tumor CoronaHack Aptos MESSIDOR-2

Metrics: accuracy average response length

Baselines: QoQ-Med LingShu MedVLThinker HuatuoGPT-Vision MedGemma-27B GPT-4o DeepSeek-R1 InternVL3.5-8B GLM-4.1V-9B LLaVA-Med PubMedVision

Builds on: DeepSeek-R1 distillation S1 (16-gram decontamination) OpenThoughts II-Medical LLaMA-Factory

Closest papers by meaning

• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.91)
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.91)
• Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.90)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.89)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.89)
• MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis (CVPR 2026 · similarity 0.88)