Research map / Large language models / Reasoning and chain of thought

AnesSuite: A Comprehensive Benchmark and Dataset Suite for Anesthesiology Reasoning in LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AnesSuite (AnesBench benchmark plus AnesCorpus, AnesQA, AnesR1 training sets) for anesthesiology reasoning, and Morpheus Qwen2.5-based models trained with SFT and GRPO on AnesR1.

Problem

LLM reasoning in specialized medical subfields like anesthesiology is underexplored; existing benchmarks mostly test factual recall and lack training resources for complex clinical decision-making.

Approach

A tiered (System 1 / 1.x / 2) bilingual anesthesiology benchmark paired with CPT, SFT and RLVR datasets, and Morpheus models trained by SFT then GRPO with outcome-based binary rewards on verifiable anesthesiology MCQs with CoT.

Key results

• On AnesBench, Gemini-2.5-Pro scores 0.86 (English) and 0.85 total average, the top model; DeepSeek-R1 reaches 0.82 English total.
• Morpheus-7B (SFT+GRPO) improves average from 0.54 (SFT only) to 0.63, roughly matching Qwen2.5-14B-Instruct (0.64); Morpheus-32B reaches 0.72 average, comparable to Qwen2.5-72B-Instruct (0.70).
• Continued pre-training on AnesCorpus raises English max accuracy from 49.7% to 51.2% but degrades Chinese performance (65.9% to 60.0%).

Surprising findings

• Continued pre-training on AnesCorpus lowered Chinese benchmark accuracy, suggesting catastrophic forgetting from monolingual domain CPT.
• Medical-specific LLMs (e.g., HuatuoGPT-o1) did not show a significant advantage over recent general-purpose reasoning models on AnesBench.

Limitations

• System 2 questions are built from abstract scenarios rather than real clinical cases.
• Text-only; no multimodal clinical data.
• Morpheus trained for limited steps (40-50 GRPO steps), with LoRA for 32B due to compute.

Experimental setup

Datasets: AnesCorpus AnesQA AnesR1 Medical-o1 Fineweb Chinese Fineweb PubMed

Benchmarks: AnesBench MMLU MedQA CAB

Metrics: accuracy

Baselines: HuatuoGPT-o1 BioMistral DeepSeek-R1 DeepSeek-V3 GPT-4o Gemini-2.5-Pro Claude-3.7-Sonnet Qwen2.5-Instruct

Builds on: GRPO DeepSeek-R1 Qwen2.5 Verl LLaMA-Factory

Closest papers by meaning

• ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room (NeurIPS 2026 · similarity 0.87)
• Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.86)
• MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.85)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.85)
• Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving (ICML 2025 · similarity 0.84)
• OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning (CVPR 2026 · similarity 0.84)