AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models

Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Joshua P Gardner, Rohan Taori, Ludwig Schmidt

NeurIPS 2023 · Vision-language and multimodal › Multimodal LLMs · approach: Visual instruction tuning · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use instruction-conditioned captions instead of BLIP2 captions for GPT-4 responses (ablation)GPT-4 captioning pipeline · 150 single-image instancesinstruction-following success: 31% -> 91%Improved
multi-image instead of single-image tasksVisIT-Bench data collection · annotation yieldGPT-4 correct (%): 91.5 -> 63.0Hurt
use reference-free GPT4 metric instead of BERTScoreevaluation metric · pairwise human agreement, 5/5 annotators agreeagreement with human majority: 80% -> 93%Improved
use reference-free GPT4 metric instead of length baselineevaluation metric · pairwise human agreement, 5/5 annotators agreeagreement with human majority: 60% -> 93%Improved
use reference-backed GPT4 metric instead of reference-freeevaluation metric · pairwise human agreementagreement with human majorityNo clear effect
compare GPT-4 automatic ranking to human rankingGPT4-no-ref Elo · 7 systems with human ratingsrank correlation: rho = 1.0Improved
rank models by Elo versus win-rate vs. referenceLlamaAdapter-v2 (7B) vs LLaVA (13B) · human eval single imageElo / win-rate: 1061 vs 1085 Elo; 27.41% vs 26.23%Mixed
use GPT-4 judge instead of human judge for win-rate vs. referenceLLaVA (13B) · single imagewin-rate vs. reference: 26.23% -> 17.81%Hurt
compare human-verified GPT-4 reference to best modelLLaVA (13B) · single image human EloElo: 1223 -> 1085Hurt
use LLaVA fine-tuned on 150K instruction dataLLaVA (13B) · single image human evalElo: 786 -> 1085Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy