Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning

On the Out-Of-Distribution Generalization of Large Multimodal Models

CVPR 2025 · Label-efficient and robust learning › Zero-/few-shot and meta-learning · approach: Transductive and semantic-dictionary zero-shot · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates zero-shot OOD generalization of 15 LMMs across synthetic, natural-shift, and medical/molecular datasets, diagnoses mapping deficiency as the main cause, and shows in-context learning helps but is vulnerable to label and spurious-correlation shifts.

Problem

Generalization boundaries of large multimodal models under out-of-distribution shifts and in specialized domains (medical, molecular) where zero-shot performance is unreliable.

Approach

Systematically evaluates LMMs zero-shot on OOD and domain-specific datasets, tests three failure hypotheses (semantic misinterpretation, visual feature insufficiency, mapping deficiency), and shows in-context examples (including from shifted domains) improve performance while label and spurious-correlation shifts in ICE hurt it.

Key results

• Zero-shot LMMs match or exceed prior SOTA on PACS, VLCS, OfficeHome, DomainNet and NICO++, but all LMMs are near-random on medical and molecular datasets (e.g., Camelyon17, DrugOOD).
• In-context examples from the target distribution raise GPT-4 accuracy on XCOVFour from 0.453 to 0.843 with 8 examples, and Gemini on iWildCam from 0.975 to 0.978 with ICE.
• Label shifts and spurious correlation shifts in ICE degrade performance (e.g., GPT-4 on CMNIST drops from 0.896 zero-shot to 0.490 with 2 biased ICE), while domain shifts in ICE still substantially improve over zero-shot.

Measured effects reported in the paper

Surprising findings

• Scaling CLIP to larger ViT sizes does not consistently improve OOD performance and even declines on TerraInc and SVIRO.
• Adding detailed domain-specific prompt context yields negligible or negative gains on medical and molecular tasks.

Limitations

• Evaluation relies on closed API models (GPT-4, Gemini, Claude-3) whose training data is unknown.
• Molecular and medical ICL gains are absent, suggesting ICL is not universally effective for specialized knowledge.
• Scaling analysis uses only CLIP because other LMM sizes are not publicly available.

Experimental setup

Datasets: CMNIST RMNIST PACS VLCS OfficeHome TerraInc DomainNet Fmow iWildCam SVIRO NICO++ Camelyon17

Benchmarks: CMNIST RMNIST PACS VLCS OfficeHome TerraInc DomainNet Fmow iWildCam SVIRO NICO++ Camelyon17

Metrics: accuracy

Baselines: SOTA domain generalization methods GPT-4 Gemini-1.5 Claude-3 CLIP

Builds on: CLIP in-context learning

Closest papers by meaning

• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.88)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs (NeurIPS 2024 · similarity 0.87)
• Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap (ICLR 2025 · similarity 0.87)
• ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (ICML 2026 · similarity 0.87)
• Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.87)
• On the generalization capacity of neural networks during generic multimodal reasoning (ICLR 2024 · similarity 0.86)