Research map / Large language models / Agents, code and math
Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HDCEval, a hierarchical divide-and-conquer framework where specialized MedLlama2-7B evaluator models per medical evaluation aspect are trained with ADTO (reward-token preference optimization) on a 13,452-sample preference dataset to better align with physicians.
Problem
LLM-based evaluators and traditional metrics are misaligned with human physicians and give oversimplified assessments when evaluating freestyle medical responses from medical LLMs.
Approach
Decompose medical evaluation into primary aspects (REL, COR, EXP) and subtasks, assign each to a separate expert evaluator trained with Attribute-Driven Token Optimization, which conditions a DPO-style preference loss on aspect-specific reward tokens and freezes the first 24 layers.
Key results
- HDCEval reports a 23.92% improvement in consistency with human evaluation over PandaLM.
- On the imaging examination scenario, HDCEval reaches 84.87% REL and 79.49% COR pairwise accuracy, matching or exceeding GPT-4 (84.87%, 61.54%).
- Ablation: removing reward tokens lowers average score from 75.13 to 72.01 and removing preference data lowers it to 73.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add reward tokens during training (ablation) | HDCEval · Fine-grained medical evaluation | Evaluation accuracy: 75.13 -> 72.01 | Helped |
| Use preference data with ADTO rather than only SFT (ablation) | HDCEval · Fine-grained medical evaluation | Evaluation accuracy: 75.13 -> 73.13 | Helped |
| Train with preference data | HDCEval · Patient Question Relevance (CONT, COND, CONC) | Position and verbosity bias: Position Bias: CONT 32.91 -> 28.57; COND 31.82 -> 26.50; CONC 32.91 -> 29.87; Verbosity Bias: CONT 32.96 -> 28.17; COND | Helped |
| Use different input formats: jointly evaluate two responses versus evaluate separately then compare | HDCEval · Medical evaluation | Evaluation results | No clear effect |
| Use HDCEval instead of PandaLM | Medical evaluation across scenarios · Medical evaluation | Pairwise accuracy: 23.92% | Helped |
Surprising findings
- Removing preference data and using only SFT lowers accuracy by only 0.5% in the reported ablation, suggesting limited gains from preference data beyond ADTO.
Limitations
- Evaluation test set and dataset are constructed by the authors and annotated by a small group of doctors, limiting external validity.
- Human-evaluator comparisons rely on a limited number of doctor annotations and win-tie-lose studies.
- Reliance on AI-assisted annotation for the 13,452-sample dataset.
Experimental setup
Datasets: medical meadow wikidoc MedBench MedText MedDialog MedMCQA
Benchmarks: HDCEval medical evaluation test set
Metrics: pairwise accuracy reference match pearson correlation intraclass correlation coefficient win-tie-lose
Baselines: GPT-4 ChatGPT PandaLM MedLlama2 BSM
Builds on: direct preference optimization BSM PandaLM FlashAttention Fully Sharded Data Parallel
Closest papers by meaning
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.89)
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations (NeurIPS 2024 · similarity 0.89)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.87)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.86)
- Reliable and Diverse Evaluation of LLM Medical Knowledge Mastery (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.