Research map / Vision-language and multimodal / Multimodal LLMs

FETAL-GAUGE: A BENCHMARK FOR ASSESSING VISION-LANGUAGE MODELS IN FETAL ULTRASOUND

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Fetal-Gauge, a 42k-image, 93k-QA MCQ benchmark built from 13 public fetal ultrasound datasets to evaluate 15 VLMs across five tasks, finding the best zero-shot model (GPT-5) reaches only 55% accuracy.

Problem

No standardized benchmark exists for evaluating vision-language models on fetal ultrasound interpretation, a domain with operator dependency and few public datasets.

Approach

Unifies heterogeneous public fetal ultrasound datasets (classification, segmentation, detection labels) into a multiple-choice VQA benchmark spanning plane identification, view conformity, fetal orientation, diagnosis, and bounding-box visual grounding, and evaluates general and medical VLMs plus LoRA fine-tuned variants.

Key results

• GPT-5 achieves the highest overall accuracy at 55%, versus Lingshu-32B at 46% and most other models near random chance (26%).
• After LoRA fine-tuning, Llama-3.2-11B-Vision-Instruct reaches 85% overall accuracy (up from 33%), while Qwen2.5-VL-7B reaches 52% at 5 epochs.
• Models perform best on large bounding boxes (GPT-5 85%, Lingshu-7B 82%) but drop on small (best 48%) and medium structures (best 67%).

Surprising findings

• Fine-tuned Llama-3.2-11B reaches 85% overall accuracy, while the zero-shot best model reaches only 55%, and most models perform at chance on clinical view conformity and fetal orientation tasks.
• Models perform poorly on phantom images, most near or below random guess, despite phantoms being standard training tools.

Limitations

• Multiple-choice format may not reflect free-text clinical reasoning.
• Medical VLMs and GPT-5 training data are not disclosed, so causes of performance differences are speculative.
• Fine-tuning results rely on a single training split and include unexplained drops at some epochs (e.g., Llama at 10 epochs on CD).

Experimental setup

Datasets: 3-Vessel View Dataset FASS Echo (1st Trimester) Echo (2nd Trimester) Fetal Planes Fetal Planes & Organs Fetus Head Tumor FOCUS FPUS23 Large Fetal Head Biometry MFUP NatalIA

Benchmarks: Fetal-Gauge

Metrics: accuracy

Baselines: Lingshu-7B Lingshu-32B MedVLM-R1 MedGemma-4b-it MedGemma-27b-it HuatuoGPT-Vision-7B InternVL3-8B-Instruct Qwen2.5-VL-7B-Instruct

Builds on: VQA-Med VQA-RAD SLAKE PMC-VQA OmniMedVQA

Closest papers by meaning

• U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding (ICLR 2026 · similarity 0.89)
• F$^2$-Assist: Multi-Phase Fetal Growth Forecast and Report Generation from Ultrasound Examination (CVPR 2026 · similarity 0.89)
• MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformer (AAAI 2025 · similarity 0.88)
• Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.85)
• OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.85)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.85)