Research map / Vision-language and multimodal / Multimodal LLMs

U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces U2-BENCH, a 7,241-case ultrasound benchmark with 8 clinically inspired tasks spanning classification, detection, regression, and text generation, and evaluates ~20-23 LVLMs including the proprietary Dolphin-V1 and reports U2-Score.

Problem

Large vision-language models have been evaluated extensively on natural and general medical images, but their ability to understand ultrasound (noisy, operator-dependent, dynamic anatomy) has not been systematically benchmarked.

Approach

A unified, patient-level sampled ultrasound benchmark aggregating 40 licensed datasets into 8 tasks with task-specific prompts, uniform metrics (accuracy, RMSE, BLEU/ROUGE/BERTScore), and a case-weighted aggregate U2-Score, plus a McNemar test on whether anatomy names in prompts help.

Key results

• Dolphin-V1 achieves the highest U2-Score of 0.5835, versus 0.3250 for GPT-5 and 0.2968 for Gemini-2.5-Pro-Preview.
• No model exceeds 0.160 accuracy on keypoint detection, and all models fall below 7.5 BLEU on report generation (open-source best is DeepSeek-VL2 at 7.47).
• Anatomy names in the prompt improved Gemini-2.0-Pro-Exp accuracy from 45.1% to 52.4% on 521 breast/thyroid studies (McNemar p=6.2e-5).

Surprising findings

• Medium-sized medical models such as MedGemma-4B-it achieve the second-best CVE RMSE, outperforming larger general models on that task.
• Scaling Qwen-2.5-VL from 3B to 72B yields diminishing returns on language generation and spatial tasks.

Limitations

• Detection tasks were simplified into 9-class position classification because models failed to output valid bounding boxes.
• Dolphin-V1 is a proprietary model from the authors' affiliated group, and its evaluation is not fully reproducible.
• Static frame-based inputs only; no video or longitudinal evaluation.

Experimental setup

Datasets: 40 licensed ultrasound datasets (incl. CAMUS, FETAL PLANES DB, DDTI, FPUS23, Chinese Ultrasound Report Dataset)

Benchmarks: U2-BENCH

Metrics: accuracy F1 RMSE MAE %_tol BLEU-4 ROUGE BERTScore U2-Score

Baselines: MiniGPT-Med MedDr MedGemma-4B-it Lingshu-7B Qwen-2.5-VL-72B-Instruct DeepSeek-VL2 GPT-4o Gemini-2.5-Pro-Preview

Builds on: VLMEvalKit GMAI-MMBench

Closest papers by meaning

• FETAL-GAUGE: A BENCHMARK FOR ASSESSING VISION-LANGUAGE MODELS IN FETAL ULTRASOUND (ICLR 2026 · similarity 0.89)
• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.88)
• MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.87)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.87)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
• Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks (ICLR 2025 · similarity 0.87)