AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz

ICCV 2023 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 54 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune BLIP2 FlanT5-XXL instead of zero-shotBLIP2 FlanT5-XXL · WHOOPS! explanation-of-violationhuman rating: 0 -> 27Improved
fine-tune BLIP2 FlanT5-XXL instead of zero-shotBLIP2 FlanT5-XXL · WHOOPS! identify weird imagesbinary accuracy: 50 -> 73Improved
scale fine-tuned BLIP2 from FlanT5-XL to XXLBLIP2 fine-tuned · WHOOPS! explanation-of-violationhuman rating: 15 -> 27Improved
give GPT3 the ground-truth caption (oracle) instead of the predicted caption (ablation)Caption -> GPT3 pipeline · WHOOPS! explanation-of-violationhuman rating: 33 -> 68Improved
replace GPT3 with GPT4 or Llama-2 for the predicted-caption pipelineCaption -> LLM pipeline · WHOOPS! explanation-of-violationGPT4 rating: 36 -> 36No clear effect
remove image input (text-only fine-tuning) (ablation)BLIP2 FlanT5-XXL · WHOOPS! image captioningCIDEr: 177 -> 2Hurt
remove image input (text-only fine-tuning) (ablation)BLIP2 FlanT5-XXL · WHOOPS! cross-modal matchingspecificity accuracy: 84 -> 94Improved
caption weird images versus normal/natural images (ablation)BLIP2 zero-shot captioning · WHOOPS! caption accuracy, 300 images (human rated)caption accuracy: 89% -> 49%Hurt
filter out VQA items answerable by text-only FlanT5 XL (ablation)WHOOPS! VQA construction · WHOOPS! VQAtext-only BEM: removed about 30% of questionsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy