Research map / Vision-language and multimodal / Human and embodied modeling

NOVA: A Benchmark for Rare Anomaly Localization and Clinical Reasoning in Brain MRI

NeurIPS 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Medical vision-language models · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces NOVA, an evaluation-only benchmark of 906 brain MRI scans across 281 rare pathologies with bounding-box, captioning, and diagnostic reasoning tasks, and benchmarks GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B on it.

Problem

Existing medical anomaly benchmarks use fixed known abnormality categories, so evaluation collapses to closed-set and does not test rare or truly novel brain MRI pathologies or the full diagnostic workflow (localization, description, diagnosis).

Approach

An evaluation-only brain MRI benchmark built from real rare-disease Eurorad cases with double-read radiologist bounding boxes, clinical narratives, and captions, designed as a zero-shot stress test of OOD generalization across three linked tasks.

Key results

• Localization on NOVA is poor: Qwen2.5-VL-72B reaches mAP30 37.66 and mAP50 24.49, with Gemini 2.0 Flash lowest at mAP50 7.37.
• GPT-4o achieves the best diagnostic reasoning with 24.2% Top-1 and 38.4% Top-5 accuracy.
• Model diagnosis outputs collapse to roughly 30% of the ground-truth label space, and entropy drops by about 1 bit (ground truth 8.68 bits).

Measured effects reported in the paper

Surprising findings

• Qwen2.5-VL-72B is the strongest localizer yet underperforms on captioning metrics, while Gemini 2.0 Flash leads captioning but is weakest at localization.
• Models produce more diverse sentences than ground truth yet have far smaller vocabularies, indicating verbose redundancy.

Limitations

• Sourced from a single European teaching repository (Eurorad), which may introduce geographic or demographic bias.
• Provides only 2D slices rather than 3D volumes.
• Proprietary models may have seen Eurorad cases during training, so results may be an upper bound on zero-shot generalization.

Experimental setup

Datasets: Eurorad neuroradiology cases (906 brain MRI scans, 281 diagnoses)

Benchmarks: NOVA anomaly localization NOVA image captioning NOVA diagnostic reasoning

Metrics: mAP@30 mAP@50 mAP@[50:95] ACC50 TP30 FP30 Clinical Term F1 Modality Term F1 BLEU METEOR

Baselines: Gemini 2.0 Flash GPT-4o Qwen2-VL-72B Qwen2.5-VL-72B

Builds on: MOOD fastMRI+ BraTS

Closest papers by meaning

• OmniBrainBench: A Comprehensive Multimodal Benchmark for Brain Imaging Analysis Across Multi-stage Clinical Tasks (CVPR 2026 · similarity 0.87)
• Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks (ICLR 2026 · similarity 0.87)
• Unraveling Normal Anatomy via Fluid-Driven Anomaly Randomization (CVPR 2025 · similarity 0.86)
• Enhancing vision-language models for medical imaging: bridging the 3D gap with innovative slice selection (NeurIPS 2024 · similarity 0.85)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.85)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.85)