Research map / Vision-language and multimodal / Multimodal LLMs
HEMM: Holistic Evaluation of Multimodal Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HEMM, a holistic benchmark of 30 image-text datasets organized by basic multimodal skills, information flow, and real-world use cases, and evaluates 11 multimodal foundation models with BARTScore plus human pairwise preference.
Problem
Existing multimodal benchmarks are collections of task-specific datasets and do not systematically characterize what multimodal foundation models can do or which modeling and training decisions help.
Approach
A three-level taxonomy (basic multimodal skills, information flow, real-world use cases) over 30 image-text datasets, paired with a categorization of models by size, pretraining data scale/diversity, training type, instruction tuning, and modality processing, used to analyze which factors drive performance.
Key results
- Gemini achieves the highest average BARTScore (0.44) and Elo rating (1074), with GPT-4V close behind (0.40, Elo 1057).
- Large models average 0.42 BARTScore versus 0.24 for medium and 0.23 for small; instruction-tuned models average 0.30 versus 0.22 without instruction tuning.
- Healthcare, HCI, and science use cases score lowest, with Decimer (0.07), iNaturalist (0.08), and Enrico (0.12) among the hardest datasets; OpenFlamingo (0.06) and Emu (0.11) rank lowest.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use large rather than small or medium models | Multimodal foundation models · HEMM 30 datasets | BARTScore: 42.33 vs 23.87 and 23.34 | Helped |
| Increase model size | MM-IMDb · MM-IMDb | BARTScore: 0.15 -> 0.36 -> 0.48 | Helped |
| Use medium rather than small pretraining data | Multimodal foundation models · HEMM 30 datasets | BARTScore: 30.10 vs 16.80 | Helped |
| Use diverse rather than non-diverse pretraining data | Multimodal foundation models · HEMM 30 datasets | BARTScore: 30.15 vs 21.71 | Helped |
| Use diverse rather than non-diverse pretraining data | Multimodal foundation models · iNaturalist | BARTScore: 0.08 vs 0.24 | Helped |
| Add instruction tuning | Multimodal foundation models · HEMM 30 datasets | BARTScore: 29.71 vs 22.49 | Helped |
| Add instruction tuning | Multimodal foundation models · Decimer | BARTScore: 0.15 | Helped |
| Use separate rather than interleaved modality inputs | Multimodal foundation models · HEMM 30 datasets | BARTScore: 28.58 vs 22.94 | Helped |
| Use modular rather than end-to-end training | Multimodal foundation models · HEMM 30 datasets | BARTScore: 24.92 vs 21.26 | Helped |
| Require external knowledge rather than no external knowledge | HEMM tasks · HEMM dataset dimensions | BARTScore: 23.51 vs 29.62 | Hurt |
| Use translation rather than fusion information flow | HEMM tasks · HEMM dataset dimensions | BARTScore: 18.97 vs 33.77 | Hurt |
| Require fine-grained rather than coarse-grained alignment | HEMM tasks · HEMM dataset dimensions | BARTScore: 26.52 vs 27.52 | No clear effect |
Surprising findings
- Small Instruct-BLIP and BLIP-2 rank close to GPT-4V and Gemini despite far fewer parameters.
- GPT-4V often refuses or answers with 'Indeterminate' or 'Unknown' on some datasets, lowering its BARTScore relative to its Elo rating.
Limitations
- Evaluation covers only a subset of possible skills, information flows, and use cases.
- Does not yet measure fairness, robustness, social bias, privacy, or efficiency.
- GPT-4V and Gemini were evaluated on only 100 samples per dataset due to query limits.
Experimental setup
Datasets: VQA Visual Genome VCR OK-VQA GQA NoCaps Flickr30K Winoground NLVR NLVR2 IRFL MM-IMDb
Benchmarks: HEMM
Metrics: BARTScore min-max normalized score win rate Elo rating
Baselines: GPT-4V Gemini Instruct-BLIP BLIP-2 mPLUG-Owl LLaMA-Adapter Fuyu-8B Mini-GPT-4 Kosmos-2 Emu OpenFlamingo
Builds on: BARTScore BLIP-2 Instruct-BLIP Flamingo
Closest papers by meaning
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.87)
- Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.87)
- MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.87)
- CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
- Unison: Benchmarking Unified Multimodal Models via Synergistic Understanding and Generation (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.