Research map / Architectures and efficiency / Mixture of experts
Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 12-language medical SFT dataset and proposes Post-MoE (sparse MoE only in last layers, with Hybrid-k cross-lingual routing) plus language-family experts, yielding Apollo-MoE that scales medical LLMs to 50 languages.
Problem
Medical LLMs lack coverage for low-resource languages due to scarce medical data, and per-language experts in MoE scale parameters linearly with the number of languages.
Approach
A Hybrid-k routing that forces language-specific experts to be activated alongside top-k cross-lingual routing, applied only in the last layers (Post-MoE) based on a "Spread Out in the End" circuit analysis, with experts grouped by linguistic family so the expert count does not grow with languages.
Key results
- Apollo-MoE 8.02B (active) reaches 69.9 average accuracy on high-resource and 58.3 on low-resource languages, leading open-source baselines such as Llama3-8B (49.3 / 33.3).
- Hybrid-k routing outperforms Top-k in low-resource languages for Qwen2-7B (58.3 vs 56.7 average) across 38 low-resource languages.
- Post-MoE with MoE in the last two layers gives the best balance of accuracy and multilingual generalization on the 38-language evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove math and code training data (ablation) | Gemma-2B multilingual training · 12 high-resource languages | average accuracy: 48.6 -> 42.7 | Hurt |
| Use Hybrid-k instead of Top-k routing | Qwen2-0.5B MoE · 12 high-resource languages | average accuracy: 39.7 -> 40.0 | Helped |
| Use Hybrid-k instead of Top-k routing | Qwen2-0.5B MoE · 38 low-resource languages | average accuracy: 29.9 -> 32.0 | Helped |
| Replace language-specific routing with Hybrid-k cross-lingual routing | Qwen2-0.5B MoE · 12 high-resource languages | average accuracy: 30.9 -> 40.0 | Helped |
| Add Hybrid-k routing rather than dense training (ablation) | Qwen2-1.5B Post-MoE · 50 languages, high-resource | average accuracy: 52.2 -> 54.8 | Helped |
| Use Hybrid-k instead of Top-k routing (ablation) | Qwen2-7B Post-MoE · 50 languages, low-resource | average accuracy: 56.7 -> 58.3 | Helped |
| Use Post-MoE in the last two layers (ablation) | Qwen2-0.5B and Qwen2-1.5B · High- and low-resource multilingual evaluation | accuracy | Helped |
| Increase low-resource training examples per language (ablation) | Qwen2-0.5B Post-MoE · 50 languages, low-resource | accuracy | Helped |
| Fine-tune the dense base model on 12-language medical data | Qwen2-0.5B · 38 low-resource languages | average accuracy: 31.5 -> 24.6 | Hurt |
Surprising findings
- Dense fine-tuning on high-resource languages degrades performance on low-resource languages (e.g., Qwen-0.5B low-resource average falls from 31.5 to 24.6 after training).
- Math and code data removal reduces average accuracy by 5.9%, suggesting they act as cross-lingual anchors.
Limitations
- Base models only go up to 7B; optimal number of MoE layers for larger models may differ.
- Low-resource training data is 2,000 machine-translated examples per language and evaluation uses Google Translate, with limited human verification (100 items for two languages).
Experimental setup
Datasets: MMLU MedQA MedMCQA PubMedQA CMMLU HEAD-QA MathInstruct Alpaca ShareGPT PubMed C4
Benchmarks: MMLU medical subsets MedQA MedMCQA HEAD-QA CMMLU medical KorMedMCQA IgakuQA RuMedBench FrenchMedMCQA MedExpQA
Metrics: accuracy
Baselines: GPT-4o Claude 3 Opus GPT-4 Turbo Gemini 1.5 Pro Llama3-8B OpenBioLLM MMed-Llama-3 JSL-MedPhi2 Top-k routing Dense model
Builds on: Sparse Upcycling Mixture of Experts Circuits
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias (NeurIPS 2024 · similarity 0.87)
- Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs (AAAI 2025 · similarity 0.86)
- CareBot: A Pioneering Full-Process Open-Source Medical Language Model (AAAI 2025 · similarity 0.86)
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.85)
- PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.