Research map / Vision-language and multimodal / Multimodal LLMs
VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Joshua P Gardner, Rohan Taori, Ludwig Schmidt
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use instruction-conditioned captions instead of BLIP2 captions for GPT-4 responses (ablation) | GPT-4 captioning pipeline · 150 single-image instances | instruction-following success: 31% -> 91% | Improved |
| multi-image instead of single-image tasks | VisIT-Bench data collection · annotation yield | GPT-4 correct (%): 91.5 -> 63.0 | Hurt |
| use reference-free GPT4 metric instead of BERTScore | evaluation metric · pairwise human agreement, 5/5 annotators agree | agreement with human majority: 80% -> 93% | Improved |
| use reference-free GPT4 metric instead of length baseline | evaluation metric · pairwise human agreement, 5/5 annotators agree | agreement with human majority: 60% -> 93% | Improved |
| use reference-backed GPT4 metric instead of reference-free | evaluation metric · pairwise human agreement | agreement with human majority | No clear effect |
| compare GPT-4 automatic ranking to human ranking | GPT4-no-ref Elo · 7 systems with human ratings | rank correlation: rho = 1.0 | Improved |
| rank models by Elo versus win-rate vs. reference | LlamaAdapter-v2 (7B) vs LLaVA (13B) · human eval single image | Elo / win-rate: 1061 vs 1085 Elo; 27.41% vs 26.23% | Mixed |
| use GPT-4 judge instead of human judge for win-rate vs. reference | LLaVA (13B) · single image | win-rate vs. reference: 26.23% -> 17.81% | Hurt |
| compare human-verified GPT-4 reference to best model | LLaVA (13B) · single image human Elo | Elo: 1223 -> 1085 | Hurt |
| use LLaVA fine-tuned on 150K instruction data | LLaVA (13B) · single image human eval | Elo: 786 -> 1085 | Improved |
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models (CVPR 2026 · similarity 0.89)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.88)
- Contextualized Evaluation of Vision Language Models through Dynamic Interviews (NeurIPS 2026 · similarity 0.88)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.88)
- Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
