Research map / Large language models / Agents, code and math
M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, Lidong Bing
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| EN-Translation prompt instead of monolingual (ablation) | ChatGPT · M3Exam Thai | accuracy: 34.09 -> 46.49 | Improved |
| EN-Translation prompt instead of monolingual (ablation) | ChatGPT · M3Exam Javanese | accuracy: 37.47 -> 50.94 | Improved |
| EN-Translation prompt instead of monolingual (ablation) | ChatGPT · M3Exam Italian | accuracy: 67.94 -> 62.76 | Hurt |
| EN-Instruct prompt instead of monolingual (ablation) | ChatGPT · M3Exam (non-English) | accuracy: Thai 34.09 -> 32.70; Italian 67.94 -> 69.30 | No clear effect |
| add few-shot demonstrations (ablation) | ChatGPT · M3Exam (non-English) | accuracy: Vietnamese 57.18 -> 58.64; Javanese 37.47 -> 33.42 | Mixed |
| GPT-4 vs ChatGPT | GPT-4 · M3Exam avg | accuracy: 57.57 -> 72.92 | Improved |
| BLIP-2 vision input vs text-only Flan-T5 | BLIP-2 · M3Exam English image questions | accuracy: 48.30 -> 49.06 | No clear effect |
| multi-image pretrained models (Fromage, OpenFlamingo) vs single-image BLIP-2 | multimodal LLMs · M3Exam English image questions | accuracy: 49.06 -> 29.81 | Hurt |
| increase education level (low to high) | LLMs · M3Exam | accuracy | No clear effect |
Closest papers by meaning
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.91)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.91)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.91)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.90)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.90)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
