Research map / Large language models / Agents, code and math
MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MTBBench, a longitudinal multimodal oncology benchmark simulating molecular tumor board workflows, evaluates LLMs/VLMs with and without foundation-model and knowledge tools.
Problem
Existing clinical benchmarks are static, unimodal, single-turn QA and do not capture the longitudinal, multimodal, multi-agent decision-making of molecular tumor boards in oncology.
Approach
A multi-turn agentic benchmark where agents must request modality-specific files (non-persistent across turns) from head-and-neck (H&E, IHC, hematology) and MSK longitudinal oncology cases, augmented with pathology foundation-model tools (CONCH, UNI2-based ABMIL IHC tool), PubMed, and DrugBank.
Key results
- Best no-tool multimodal overall accuracy was InternVL 78B at 69.1%, 2.5 points above GPT-4o.
- Longitudinal outcome accuracy reached 83.0% for Qwen3 32B, while recurrence and progression stayed near chance.
- Tool augmentation (CONCH, IHC ABMIL tool, PubMed, DrugBank) improved digital pathology accuracy by up to 9% and longitudinal progression/recurrence by over 5% in selected models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual foundation model tools | LLM agents on MTBBench-Multimodal · MTBBench-Multimodal | accuracy: up to 9.0% | Helped |
| Add PubMed and DrugBank tools | LLM agents on MTBBench-Longitudinal · MTBBench-Longitudinal | accuracy: up to 11.2% | Helped |
| Access more patient files and modalities | Agents on MTBBench · MTBBench multimodal and longitudinal tasks | accuracy | Helped |
Surprising findings
- Outcome and recurrence prediction stayed near random (~50%) even for leading models.
- Model size did not reliably predict pathology accuracy; Gemma 3 12B outperformed Gemma 3 27B.
Limitations
- Offline, controlled benchmark; agents not tested in interactive real clinical workflows.
- Small patient cohorts (26 multimodal, 40 longitudinal patients).
- Longitudinal track has no specialized foundation model tools, so tool gains are modest.
Experimental setup
Datasets: HCC head and neck multimodal dataset (26 patients) MSK clinicogenomic dataset (40 patients)
Benchmarks: MTBBench-Multimodal MTBBench-Longitudinal
Metrics: accuracy bootstrap 95% confidence intervals average files accessed per question
Baselines: GPT-4o o4-mini Gemma-3 27B Llama 3.3 70B Qwen2.5-VL 32B InternVL 78B
Builds on: CONCH UNI2 ABMIL TRIDENT
Closest papers by meaning
- MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.88)
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.88)
- Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.88)
- MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.88)
- MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks (NeurIPS 2025 · similarity 0.88)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.