Research map / Large language models / Retrieval and knowledge

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge-augmented question answering · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Medmarks, a fully open-source medical LLM benchmark suite of 30 benchmarks (verifiable and LLM-as-judge open-ended subsets) evaluating 61 models across 71 configurations, with verifiers environments usable for RL post-training.

Problem

Existing medical LLM benchmarks are saturated, depend on restricted datasets, or lack comprehensive task and model coverage, so there is no regularly updated, fully open, easy-to-run medical evaluation suite.

Approach

A fully open suite of 30 medical benchmarks split into verifiable (Medmarks-V) and LLM-as-a-judge open-ended (Medmarks-OE) subsets, implemented as verifiers environments with reward functions, a robust multiple-choice answer parser, dataset-weighted mean win rate aggregation, and answer-order shuffling analysis; the trainable subset (Medmarks-T) doubles as RL environments.

Key results

• Gemini 3 Pro Preview achieves the top Medmarks-V mean win rate (0.6628), followed by GPT-5.1 (med) at 0.6395 and Grok 4 at 0.6343.
• On Medmarks-OE, GPT-5.2 (med) leads with 0.6389 win rate, while Gemini 3 Pro Preview drops to 0.4713.
• Medical fine-tunes such as MedGemma 27B (0.502 vs Gemma 3 27B 0.461 win rate) and AntAngelMed (0.552 vs Ling Flash 2.0 0.517) outperform their generalist bases on average.

Surprising findings

• Baichuan M3 235B outperforms Qwen3 235B-A22B Thinking on Medmarks-OE (0.5678 vs 0.5161) while being cheaper to run.
• Smaller models and Grok 4 show notable answer-order sensitivity; Grok 4's score variance on M-ARC drops from 0.11 to 0.02 when answer order is held fixed.

Limitations

• Benchmark is heavily biased toward single-turn question answering; agentic coverage is limited.
• Limited evaluation of fairness, bias, and safety.
• Text-only; does not evaluate multimodal medical capabilities.

Experimental setup

Datasets: MedQA MedMCQA PubMedQA MedCalc-Bench MeQSum MEDEC MedDialog MedCaseReasoning MedConceptsQA HealthBench MedXpertQA LongHealth

Benchmarks: MedQA MedMCQA PubMedQA MedConceptsQA HEAD-QA v2 MedXpertQA MedCalc-Bench LongHealth Med-HALT MedHallu MMLU-Pro-Health M-ARC

Metrics: accuracy mean win rate LLM-as-a-judge score token count cost

Baselines: MedHELM MultiMedQA HealthBench MedS-Bench ClinicBench MedAgentsBench

Builds on: MedHELM HealthBench verifiers MultiMedQA

Closest papers by meaning

• MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.93)
• MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.92)
• MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.90)
• Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity (ICML 2025 · similarity 0.90)
• Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation (AAAI 2025 · similarity 0.89)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.89)