Research map / Vision-language and multimodal / Multimodal LLMs
EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EndoBench, a benchmark of 6,832 multiple-choice VQA pairs across 4 endoscopy scenarios, 12 clinical tasks and 5 visual prompt granularities, evaluating 23 MLLMs against clinician performance.
Problem
Existing endoscopy VQA benchmarks cover only specific scenarios and few clinical tasks, so they fail to measure whether MLLMs handle the full range of endoscopic practice and skills.
Approach
A comprehensive endoscopy MLLM benchmark spanning gastroscopy, colonoscopy, capsule and surgical endoscopy, organized around clinical workflow categories, with multiple visual prompt types (image, box, contour, multi-box, multi-contour, coordinate) and human clinician reference scores.
Key results
- Gemini-2.5-Pro achieves the highest average accuracy of 49.53% across 12 tasks, versus 74.12% for physicians.
- GPT-4o reaches 41.69% average; medical-domain HuatuoGPT-Vision-34B reaches 39.58%, and ColonGPT averages only 15.60%.
- For polyp counting, adding bounding-box visual prompts raises Gemini-2.5-Pro from 24.46% to 92.57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use medical-domain supervised fine-tuning | medical MLLMs · EndoBench, task-specific clinical tasks | accuracy | Helped |
| Use proprietary MLLMs rather than open-source or medical-specialized models | MLLMs · EndoBench, 12 clinical tasks | average accuracy: 49.53% (Gemini-2.5-Pro); 74.12% (Physician) | Helped |
| Use Gemini-2.5-Pro instead of physician experts | EndoBench · 12 clinical tasks | average accuracy: 74.12% ... 49.53% | Hurt |
| Add bounding boxes as visual prompts | Gemini-2.5-Pro · polyp counting | accuracy: 24.46% -> 92.57% | Helped |
| Use region-focused visual prompts | MLLMs · Region Selection | accuracy | Helped |
| Apply medical-domain supervised fine-tuning | MedDr and HuatuoGPT-Vision-34B · landmark and organ identification | accuracy | Helped |
| Use open-source MLLMs instead of random guessing | open-source MLLMs · surgical scenarios | accuracy | Helped |
| Use open-source MLLMs instead of random guessing | open-source MLLMs · landmark and organ identification | accuracy | No clear effect |
| Use current MLLMs for lesion-relevant tasks rather than surgery-related tasks | MLLMs · EndoBench clinical task categories | accuracy | Hurt |
| Use bounding-box prompts rather than the original prompting approach | MLLMs · polyp counting | accuracy | Helped |
| Use different visual prompt formats | MLLMs · EndoBench spatial understanding | accuracy | Mixed |
Surprising findings
- Medical-specialized models such as MedDr and HuatuoGPT-Vision-34B outperform proprietary models on some anatomy tasks, but ColonGPT scores only 15.60% average.
- Model accuracy changes substantially with visual prompt format, e.g., polyp counting jumps from under 30% to over 90% with bounding boxes.
Limitations
- Benchmark relies on multiple-choice format, which may not reflect open-ended clinical reasoning.
- Only two clinicians provided human reference scores on a 255-question subset.
- Includes one private in-house capsule endoscopy dataset that may not be fully accessible.
Experimental setup
Datasets: Kvasir HyperKvasir Kvasir-Capsule GastroVision KID WCEBleedGen SEE-AI Kvasir-Seg CVC-ColonDB ETIS-Larib CVC-ClinicDB CVC-300
Benchmarks: EndoBench
Metrics: accuracy
Baselines: Gemini-2.5-Pro GPT-4o Claude-3.7-Sonnet Grok-3 Deepseek-V3 MedDr HuatuoGPT-Vision LLaVA-Med ColonGPT Qwen2.5-VL InternVL2.5 Janus-Pro
Builds on: LLaVA DINOv2 ColonGPT
Closest papers by meaning
- Gastric-X: A Multimodal Multi-Phase Benchmark Dataset for Advancing Vision-Language Models in Gastric Cancer Analysis (CVPR 2026 · similarity 0.88)
- Multi-view Masked Contrastive Representation Learning for Endoscopic Video Analysis (NeurIPS 2024 · similarity 0.85)
- U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.85)
- MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.85)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.85)
- M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.