Research map / Large language models / AI and society
Multilingual Medical Language Models: A Path to Improving Lay Health Worker Effectiveness (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position-style student abstract proposing a pipeline for LLM-powered virtual assistants for lay health workers in Telugu and Hindi, covering localized data augmentation, NeMo Guardrails, and knowledge distillation into an Ayurllama model.
Problem
Lay health workers in developing nations rely on unreliable sources for medical information, and general LLMs perform poorly in low-resource languages like Telugu and Hindi in medical domains, while deployment costs and guardrail needs remain barriers.
Approach
Build a modular pipeline that starts from English medical dialog data, augments it with local cultural and linguistic elements, fine-tunes open-source medical LLMs, adds programmatic guardrails, and uses knowledge distillation to cut deployment cost.
Key results
- Ayurllama (13B) scored 57.2 on PubMedQA(ID), below LLaMA-2 13B (68.0) and ChatGPT (63.9) in the reported table.
- NeMo Guardrails reportedly reduce inference performance by 15% to 20%.
- The authors report a deployment and inference cost of $0.002 per interaction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NeMo Guardrails | LLM inference pipeline · Inference | inference performance: 15% to 20% | Hurt |
| Apply knowledge distillation | Ayurllama · Base medical dataset | performance | Helped |
Surprising findings
- Ayurllama, the distilled model, underperforms both its LLaMA-2 13B base comparison and ChatGPT on PubMedQA(ID).
Limitations
- Guardrails add a 15% to 20% impact on inference performance.
- NeMo Guardrails primarily supports English, limiting applicability to other languages.
- Ayurllama underperforms baselines on the single reported benchmark.
Experimental setup
Datasets: English medical dialog dataset (Chen et al. 2020)
Benchmarks: PubMedQA
Metrics: accuracy
Baselines: ChatGPT LLaMA-2
Builds on: NeMo Guardrails red-teaming RLHF
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.86)
- Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias (NeurIPS 2024 · similarity 0.85)
- Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments (AAAI 2025 · similarity 0.84)
- Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts (ICLR 2025 · similarity 0.83)
- MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models (NeurIPS 2024 · similarity 0.83)
- SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.