Research map / Vision-language and multimodal / Human and embodied modeling
SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, Ranjay Krishna
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace rule-based negative generation with ChatGPT generation | Compositionality benchmark construction · ARO+CREPE vs SugarCrepe hard-negative text | commonsense and grammar quality: 86% | Improved |
| Adversarially refine the generated benchmark to balance text-model score gaps | SugarCrepe · SugarCrepe | text-only attack accuracy: 0.5 | Improved |
| Train with Replace hard negatives | ViT-B/32 NegCLIP finetuning · ARO+CREPE Replace | accuracy: 77.06 -> 94.51 | Improved |
| Train with Replace hard negatives | ViT-B/32 NegCLIP finetuning · SugarCrepe Replace | accuracy: 84.76 -> 88.27 | Improved |
| Train with Swap hard negatives | ViT-B/32 NegCLIP finetuning · ARO+CREPE Swap | accuracy: 68.81 -> 94.48 | Improved |
| Train with Swap hard negatives | ViT-B/32 NegCLIP finetuning · SugarCrepe Swap | accuracy: 70.83 -> 76.21 | Improved |
| Evaluate hard-negative training on less biased SugarCrepe rather than ARO+CREPE | NegCLIP finetuning · Replace hard negatives | accuracy gain over CLIP finetuning: 17.45; 3.51 | Mixed |
| Scale pretrained CLIP models and training data | Pretrained CLIP · SugarCrepe Swap-Obj and Swap-Att | accuracy | No clear effect |
| Use object changes rather than attribute or relation changes | Pretrained CLIP models · SugarCrepe Replace | accuracy: 15%; 29% | Improved |
| Use object additions rather than attribute additions | Pretrained CLIP models · SugarCrepe Add | accuracy: 10% | Improved |
Closest papers by meaning
- SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations (NeurIPS 2024 · similarity 0.91)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.88)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.88)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.87)
- BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
