Research map / Vision-language and multimodal / Multimodal LLMs

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces FOCUS, a cross-dataset benchmark harmonizing binary DR, referable DR, and glaucoma tasks across ten fundus datasets to evaluate vision encoders, VLMs, and MLLMs (plus LoRA SFT) on performance, calibration, fairness, and quality robustness.

Problem

Single-dataset leaderboards do not reveal how retinal foundation models (vision encoders, VLMs, MLLMs) behave under dataset shift, across clinical label definitions, across patient subgroups, or under image-quality variation.

Approach

A harmonized multi-dataset, multi-interface evaluation layer that scores VM linear probes, VLM zero-shot/linear-probe, and MLLM first-token yes/no probabilities on the same tasks and four reliability axes, plus a same-task transfer analysis of LoRA SFT MLLM adapters trained on BRSET and mBRSET.

Key results

• General VM encoders reach the highest overall mean AUROC for linear probing (0.828) and the largest ophthalmic-vs-generalist gap appears on mBRSET (+0.162 for generalist VM).
• Glaucoma is hardest: generalist VM AUROC drops from 0.867 (binary DR) to 0.742 (glaucoma) and medical MLLMs drop from 0.868 to 0.656 in the same comparison.
• LoRA SFT on BRSET/mBRSET improves in-domain accuracy by +0.033 and ECE by -0.040 but AUROC changes only +0.004 (not significant).

Surprising findings

• Generalist VM encoders outperform ophthalmic-domain VM encoders on most datasets, including the mobile-phone dataset mBRSET.
• Scaling MedGemma from 4B to 27B does not improve mean AUROC, while medical specialization over Gemma-3-27B does (0.648 to 0.797).

Limitations

• Not a clinical validation study; retrospective evaluation only, with labels inherited from heterogeneous public sources.
• Some datasets lack demographic or image-quality metadata, limiting fairness and robustness analyses.
• MLLM results depend on prompt templates, answer parsing, and token-probability availability.

Experimental setup

Datasets: BRSET mBRSET PAPILA RFMiD RFMiD 2.0 IDRiD Messidor-2 REFUGE G1020 JSIEC1000

Benchmarks: FOCUS

Metrics: AUROC AUPRC accuracy F1 ECE demographic parity gap equal opportunity gap equalized odds gap image-quality robustness gap

Baselines: RETFound VisionFM FLAIR RET-CLIP EyeCLIP FunBench LMOD

Builds on: RETFound CLIP SigLIP2 MedSigLIP MedGemma LLaVA-NeXT

Closest papers by meaning

• Domain Adaptive Diabetic Retinopathy Grading with Model Absence and Flowing Data (CVPR 2025 · similarity 0.89)
• Like an Ophthalmologist: Dynamic Selection Driven Multi-View Learning for Diabetic Retinopathy Grading (AAAI 2025 · similarity 0.88)
• LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.86)
• BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · similarity 0.86)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI (NeurIPS 2024 · similarity 0.85)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.85)