Research map / Large language models / Retrieval and knowledge
Multi-OphthaLingua: A Multilingual Benchmark for Assessing and Debiasing LLM Ophthalmological QA in LMICs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Multi-OphthaLingua, a paired multilingual ophthalmology MCQ benchmark across seven languages, and CLARA, an agentic inference-time debiasing system combining translation, evaluation, RAG, web search and query rewriting.
Problem
LLMs perform unevenly across languages on medical question answering, which may worsen healthcare disparities in LMICs; no paired multilingual ophthalmology QA benchmark existed to measure this.
Approach
A manually curated, parallel multilingual ophthalmology MCQ benchmark enabling direct cross-lingual comparison, plus a multi-agent reflective pipeline (translation, meta-cognitive evaluation, weighted/jargon-aware RAG, web search, question rewriting) that narrows the language gap at inference time.
Key results
- GPT-4 direct inference reaches 63.4% in English but 51.8% in Filipino and 50.6% in Hindi.
- CLARA raises GPT-4 Filipino accuracy from 51.8% to 67.1% and English from 63.4% to 72.2% (reported in the debias table).
- On GPT-3.5, CLARA raises Filipino from 34.8% to 45.7%, narrowing the Filipino gap from -11.4 to -7.9 points relative to English (its own table, with English at 53.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CLARA instead of Direct Inference | GPT-4 · Multi-OphthaLingua FIL | accuracy: 51.8 -> 67.1 | Helped |
| apply CLARA instead of Direct Inference | GPT-4 · Multi-OphthaLingua ZH | accuracy: 52.4 -> 70.2 | Helped |
| apply CLARA instead of Direct Inference | GPT-4 · Multi-OphthaLingua HIN | accuracy: 50.6 -> 68.4 | Helped |
| apply CLARA instead of Translate-COT | GPT-4 · Multi-OphthaLingua FIL | accuracy: 61.6 -> 67.1 | Helped |
| add translation and English chain-of-thought (ablation) | GPT-4 · Multi-OphthaLingua FIL | accuracy: 51.8 -> 61.6 | Helped |
| add Web ToolCall to Translate-COT (ablation) | GPT-4 · Multi-OphthaLingua FIL | accuracy: 61.6 -> 61.8 | Helped |
| add Basic-RAG to translation and Web ToolCall (ablation) | GPT-4 · Multi-OphthaLingua FIL | accuracy: 61.8 -> 62.5 | Helped |
| add Corrective-RAG (ablation) | GPT-4 · Multi-OphthaLingua FIL | accuracy: 62.5 -> 64.9 | Helped |
| add the CLARA rewriting agent (ablation) | GPT-4 · Multi-OphthaLingua FIL | accuracy: 64.9 -> 67.1 | Helped |
| add the full CLARA system (ablation) | GPT-4 · Multi-OphthaLingua EN | accuracy: 66.4 -> 72.2 | Helped |
| use Web-ToolCall instead of Direct Inference | GPT-3.5 · Multi-OphthaLingua ZH | accuracy: 32.3 -> 31.1 | Hurt |
| use Web-ToolCall instead of Direct Inference | GPT-3.5 · Multi-OphthaLingua HIN | accuracy: 37.2 -> 37.2 | No clear effect |
Surprising findings
- Translate-CoT and CLARA gains are not uniform in direction across methods; web-search tool calls sometimes hurt performance, notably for GPT-4 on Spanish, Portuguese and French.
- Qwen-2 72B performs better on Mandarin than on English-adjacent languages, an exception to the LMIC-language pattern attributed to Chinese pretraining.
Limitations
- Benchmark has 1184 questions and results are averaged over 8 runs with temperature 0, so variance is hard to judge.
- Some reported numbers are inconsistent between tables (e.g., GPT-4 English 63.4 vs 66.4 in ablation).
Experimental setup
Datasets: PubMed Medical textbooks Wikipedia
Benchmarks: Multi-OphthaLingua
Metrics: accuracy cross-lingual accuracy gap
Baselines: Direct inference Web-ToolCall Translate-COT EN-COT Basic RAG
Builds on: Translate-COT Self-RAG MedCPT Corrective RAG
Closest papers by meaning
- MedJourney: Benchmark and Evaluation of Large Language Models over Patient Clinical Journey (NeurIPS 2024 · similarity 0.88)
- Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks (NeurIPS 2026 · similarity 0.88)
- MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models (ICLR 2025 · similarity 0.87)
- JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation (NeurIPS 2026 · similarity 0.87)
- Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (ICLR 2026 · similarity 0.86)
- MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.