Research map / Vision-language and multimodal / Multimodal LLMs
VHELM: A Holistic Evaluation of Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the HELM framework to vision-language models, standardizing 21 existing VLM datasets across 9 aspects and evaluating 22 VLMs, mostly via API models and some open-weight ones.
Problem
Existing VLM benchmarks cover only a few capabilities (mostly perception and reasoning), test small and non-overlapping model sets, and use inconsistent evaluation protocols, making holistic and fair comparison of VLMs difficult.
Approach
Apply the HELM aspect-scenario-adaptation-metric framework to VLMs: aggregate 21 existing datasets into 9 aspects (visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, safety) and run all models under standardized prompts, inference parameters and metrics.
Key results
- Efficiency-focused models (Claude 3 Haiku, Gemini 1.5 Flash) perform significantly worse than full models on the bias benchmark but not on other aspects.
- No single model excels across all aspects; GPT-4o underperforms on bias, robustness, and toxicity.
- Closed-API models significantly outperform open-weight models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use efficiency-focused models (Claude 3 Haiku or Gemini 1.5 Flash) instead of their full models (Claude 3 Opus or Gemini 1.5 Pro) | Claude 3 and Gemini 1.5 model families · VHELM bias benchmark | performance | Hurt |
| Use efficiency-focused models instead of their full models | Claude 3 and Gemini 1.5 model families · VHELM aspects other than bias | performance | No clear effect |
| Use closed-API models instead of open-weight models | VLMs evaluated in VHELM · VHELM | performance | Helped |
| Use GPT-4o across aspects | GPT-4o · VHELM aspects | leaderboard performance | Mixed |
Surprising findings
- Efficiency-focused models underperform full models on bias but not on other aspects.
- Open-weight models struggle to follow even simple instructions, suggesting benefit from instruction tuning.
Limitations
- Evaluation covers only existing datasets and a snapshot of models at launch.
- Open-weight models are evaluated with limited instruction following.
Experimental setup
Datasets: MMMU VQAv2 MM-Vet
Benchmarks: MMMU VQAv2 MM-Vet
Metrics: exact match accuracy toxicity score bias score
Baselines: GPT-4o Gemini 1.5 Pro Claude 3 Opus Qwen-VL Chat IDEFICS-instruct
Builds on: HELM HEIM
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.93)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.92)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
- UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling (NeurIPS 2024 · similarity 0.91)
- WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences (NeurIPS 2024 · similarity 0.91)
- SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.