Research map / Large language models / Reasoning and chain of thought

MedS³: Towards Medical Slow Thinking with Self-Evolved Soft Dual-sided Process Supervision

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MedS³, a self-evolving MCTS-based framework that bootstraps a small medical LLM (Llama3.1-8B) via SFT/DPO and trains a soft dual-sided process reward model, reaching state-of-the-art accuracy across 11 medical benchmarks.

Problem

Small, deployable medical language models lack robust, faithful step-level reasoning, and existing medical reasoning efforts have narrow task coverage (mostly multiple-choice), weak process supervision, and rely on proprietary teacher models.

Approach

Use MCTS with rollout-based node values on a curriculum-sampled, multi-task medical seed set to self-generate reasoning trajectories, then train a PRM with soft dual-sided labels that penalize steps which degrade node value, so that steps in correct-answer trajectories with flawed reasoning are still penalized.

Key results

• MedS³ (Llama3.1-8B with PRM, Best-of-32) averages 73.10 across 11 in-domain benchmarks, versus 66.52 for GPT-4o-mini and 64.53 for R1-Distill-Qwen32B, and +6.45 over the prior best medical model per the abstract.
• Soft dual-sided PRM improves the policy's average from 68.40 (Iter 2, no PRM) to 73.10, about +4.7 points.
• On out-of-domain tasks, MedS³ w/ PRM reaches 30.66 on MedCalc, 16.44 on MedXpert and 41.20 on rare disease confirmation, the best among compared models.

Surprising findings

• Self-consistency gains for baselines are concentrated on traditional exam benchmarks and hardly appear on clinical tasks like SEER and DDXPlus, suggesting overfitting to multiple-choice formats.
• Plain value-based PRM variant (hard single-sided labels) and PRMs initialized from SFT policy underperform the soft dual-sided PRM, showing the value of training with both positive and negative responses.

Limitations

• Evaluation is largely on close-ended tasks; the training data contains no open-ended clinical generation.
• MC rollout values are acknowledged as not the best estimate of internal step value, per the authors.
• Hallucination/interpretability assessment relies on GPT-4o scoring rather than human evaluation.

Experimental setup

Datasets: MedQA MedMCQA PubMedQA BioMRC HealthFact PubHealth HeadQA SciQ Evidence Inference MQP CIC Covid-19 Classification

Benchmarks: MedQA MedMCQA PubMedQA BioASQ MMLU (medical) BioMRC PubHealth HealthFact DDXPlus DrugDose SEER MedCalc

Metrics: exact match accuracy average accuracy hallucination/interpretability score (GPT-4o rated)

Baselines: GPT-3.5-turbo GPT-4o-mini QwQ-32B-preview R1-Distill-Qwen32B Llama3-8B Llama3.1-8B Qwen2.5-7B R1-Distill-Llama8B MedLlama3 Med42 OpenBioLLM UltraMedical3-8B

Builds on: Monte Carlo Tree Search direct preference optimization GRPO Llama 3.1

Closest papers by meaning

• Optimizing Inference-Time Compute for Medical Reasoning via Uncertainty Quantification (ICML 2026 · similarity 0.88)
• Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.87)
• MedChain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence (NeurIPS 2025 · similarity 0.87)
• MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (NeurIPS 2024 · similarity 0.87)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models (NeurIPS 2025 · similarity 0.87)