Research map / Vision-language and multimodal / Visual question answering
TCM-Ladder: A Benchmark for Multimodal Question Answering on Traditional Chinese Medicine
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TCM-Ladder, a multimodal (text, image, audio, video) QA benchmark for Traditional Chinese Medicine with over 52,000 items, evaluates nine general and five TCM-specific LLMs, proposes the Ladder-Score metric, and trains a Qwen2.5-7B reasoning model with GRPO.
Problem
Existing TCM evaluation datasets are text-based, narrow in scope, and lack a unified multimodal QA benchmark for assessing LLMs on TCM tasks.
Approach
A unified multimodal TCM QA dataset covering single-choice, multiple-choice, fill-in-the-blank, diagnostic dialogue and visual tasks, paired with Ladder-Score, a weighted combination of terminology accuracy (TermScore) and LLM-judged semantic quality (SemanticScore).
Key results
- On fill-in-the-blank, BenCao reached 0.9034 exact match, the highest among evaluated models; Ladder-base reached 0.8623.
- On diagnostic dialogue, Ladder-base achieved the highest BLEU-4 (0.0249) and ROUGE-L (0.2431), while Qwen3 had the best Ladder-Score (0.861).
- Human physicians achieved 64% top-1 accuracy on herb image retrieval, about 4 points below the best model BenCao.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a reasoning model on the TCM-Ladder training subset with GRPO | Qwen2.5-7B-Instruct (Ladder-base) · Diagnostic dialogue | BLEU-4: 0.0249 | Helped |
| Train a reasoning model on the TCM-Ladder training subset with GRPO | Qwen2.5-7B-Instruct (Ladder-base) · Diagnostic dialogue | ROUGE-L: 0.2431 | Helped |
| Use BenCao, fine-tuned from ChatGPT on TCM knowledge | ChatGPT (BenCao) · Fill-in-the-blank | Exact match accuracy: 0.9034 | Helped |
Surprising findings
- Human clinicians scored lower than the best model (BenCao) on herb image recognition, which the authors attribute to processed-form variability.
- Many general-domain LLMs, including GPT-4o, Claude 3, Kimi k1.5 and Grok 3, scored below 40% on tongue classification.
Limitations
- Dataset scale does not cover full breadth of TCM; pulse, olfaction and tongue modalities are underrepresented.
- Data primarily from Chinese clinical populations, limiting demographic diversity.
- Single run per experiment due to API cost; no error bars.
Experimental setup
Datasets: TCM-Ladder training subset National Physician Qualification Examination of China Pharmacology of Chinese Herbs iTongue tongue images publicly available pulse and audio datasets
Benchmarks: TCM-Ladder
Metrics: top-1 accuracy exact match accuracy BLEU-4 ROUGE-L METEOR BERTScore Ladder-Score
Baselines: GPT-4o GPT-4o mini Gemini 2.5 Pro Gemini 2.0 Flash Deepseek Qwen3 Claude 3 Grok 3 Kimi k1.5 BenTsao HuatuoGPT2 Zhongjing
Builds on: GRPO Qwen2.5-7B-Instruct BLEU ROUGE BERTScore
Closest papers by meaning
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.89)
- Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-World Multi-Turn Dialogue (AAAI 2024 · similarity 0.88)
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.86)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.85)
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.85)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.