Research map / Vision-language and multimodal / Visual question answering
SMMILE: An expert-driven benchmark for multimodal medical in-context learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SMMILE, an expert-curated benchmark (111 problems, 517 triplets) and SMMILE++ (1038 permuted problems) for multimodal in-context learning in medicine, evaluating 15 MLLMs in zero-shot and ICL settings.
Problem
Multimodal in-context learning in medicine is underexplored; existing medical few-shot evaluations use randomly selected examples, and no benchmark evaluates whether MLLMs can learn medical tasks from expert-designed multimodal demonstrations.
Approach
Expert-authored problems where each query is paired with expert-designed multimodal in-context examples, plus controlled perturbations (noisy examples, permuted order) to probe example quality and ordering effects such as recency bias.
Key results
- Average open-ended LLM-as-a-Judge ICL gain over zero-shot is 8.01% absolute on SMMILE; GPT-4o leads with 49.88% open-ended and 58.85% MCQA with ICL.
- On SMMILE++, Qwen2.5-VL-72B reaches 53.8% LLM-as-a-Judge accuracy with ICL and 63.2% ICL-MCQA.
- Placing an answer-matching example last rather than first can improve accuracy by up to 71%, indicating recency bias; a single noisy example reduces average performance by about 9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide in-context examples instead of zero-shot input | MLLMs on SMMILE open-ended tasks · SMMILE | LLM-as-a-Judge accuracy: 8.01% (absolute) average improvement; LLaVA-Med-7B 21.65% to 10.19% | Mixed |
| provide in-context examples instead of zero-shot input | MLLMs on SMMILE++ open-ended tasks · SMMILE++ | LLM-as-a-Judge accuracy: average relative improvement of 44.7% | Helped |
| insert one irrelevant random example into the in-context list (ablation) | MLLMs on SMMILE · SMMILE | LLM-as-a-Judge accuracy: 9.1% relative decrease from SMMILE on average | Hurt |
| insert one same-specialty but noisy example into the in-context list (ablation) | MLLMs on SMMILE · SMMILE | LLM-as-a-Judge accuracy: 9.5% relative decrease from SMMILE on average | Hurt |
| add chain-of-thought prompting (ablation) | MLLMs on SMMILE open-ended ICL · SMMILE | LLM-as-a-Judge accuracy: LLaVA-OneVision-0.5B: 21.63 ± 4.00 -> 7.23 ± 2.53; Qwen2.5-VL-7B: 29.58 ± 4.63 -> 29.95 ± 4.75 | Mixed |
| place a matching-answer example first rather than later (ablation) | nine evaluated MLLMs on SMMILE · SMMILE subset of 69 problems | LLM-as-a-Judge accuracy: absolute decrease of up to 47% | Hurt |
| place a matching-answer example last rather than earlier (ablation) | nine evaluated MLLMs on SMMILE · SMMILE subset of 69 problems | LLM-as-a-Judge accuracy: absolute improvement of up to 71% | Helped |
| increase the number of in-context examples beyond two (ablation) | evaluated MLLMs on SMMILE · SMMILE | LLM-as-a-Judge accuracy | Mixed |
Surprising findings
- Adding a single irrelevant or noisy in-context example can degrade performance by up to 9.5% relative on average.
- Medical-specific models (LLaVA-Med-7B, MedGemma-4B, MedVLM-R1) do not consistently outperform size-matched general-purpose models in ICL.
Limitations
- Limited scale (111 problems) and modality coverage (images only in current release).
- Expert contributor pool may not cover all specialties or practice settings.
- Benchmark centers on diagnosis; does not cover treatment planning, prognosis, or longitudinal reasoning.
Experimental setup
Datasets: SMMILE SMMILE++
Benchmarks: SMMILE SMMILE++
Metrics: LLM-as-a-Judge accuracy exact match MCQA accuracy expert rating bootstrap standard deviation
Baselines: Random Majority Text-only Llama 3.3 70B LLaVA-Med-7B MedGemma-4B MedVLM-R1
Builds on: Flamingo LLaVA VL-ICL Bench
Closest papers by meaning
- Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE (NeurIPS 2024 · similarity 0.91)
- Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine (AAAI 2025 · similarity 0.91)
- Med-CMR: A Fine-Grained Benchmark Integrating Visual Evidence and Clinical Logic for Medical Complex Multimodal Reasoning (CVPR 2026 · similarity 0.90)
- Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search (NeurIPS 2025 · similarity 0.90)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.89)
- OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.