Research map / Vision-language and multimodal / Multimodal LLMs
Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images
Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune BLIP2 FlanT5-XXL instead of zero-shot | BLIP2 FlanT5-XXL · WHOOPS! explanation-of-violation | human rating: 0 -> 27 | Improved |
| fine-tune BLIP2 FlanT5-XXL instead of zero-shot | BLIP2 FlanT5-XXL · WHOOPS! identify weird images | binary accuracy: 50 -> 73 | Improved |
| scale fine-tuned BLIP2 from FlanT5-XL to XXL | BLIP2 fine-tuned · WHOOPS! explanation-of-violation | human rating: 15 -> 27 | Improved |
| give GPT3 the ground-truth caption (oracle) instead of the predicted caption (ablation) | Caption -> GPT3 pipeline · WHOOPS! explanation-of-violation | human rating: 33 -> 68 | Improved |
| replace GPT3 with GPT4 or Llama-2 for the predicted-caption pipeline | Caption -> LLM pipeline · WHOOPS! explanation-of-violation | GPT4 rating: 36 -> 36 | No clear effect |
| remove image input (text-only fine-tuning) (ablation) | BLIP2 FlanT5-XXL · WHOOPS! image captioning | CIDEr: 177 -> 2 | Hurt |
| remove image input (text-only fine-tuning) (ablation) | BLIP2 FlanT5-XXL · WHOOPS! cross-modal matching | specificity accuracy: 84 -> 94 | Improved |
| caption weird images versus normal/natural images (ablation) | BLIP2 zero-shot captioning · WHOOPS! caption accuracy, 300 images (human rated) | caption accuracy: 89% -> 49% | Hurt |
| filter out VQA items answerable by text-only FlanT5 XL (ablation) | WHOOPS! VQA construction · WHOOPS! VQA | text-only BEM: removed about 30% of questions | Improved |
Closest papers by meaning
- Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models (NeurIPS 2024 · similarity 0.84)
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.84)
- SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports (ICLR 2026 · similarity 0.84)
- Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models (ICML 2025 · similarity 0.83)
- Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events (CVPR 2025 · similarity 0.83)
- NL-Eye: Abductive NLI For Images (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
