Research map / Vision-language and multimodal / Human and embodied modeling
SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SugarCrepe++, a 3-way semantic (in)equivalence benchmark (two lexically different positive captions plus one hard negative per image) and evaluates many VLMs and unimodal text encoders in image-to-text and text-only settings.
Problem
Existing compositionality benchmarks such as SugarCrepe use only one positive and one lexically similar hard negative, so they cannot separate semantic understanding from lexical overlap; the sensitivity of VLM and ULM encoders to lexical variation of semantically equivalent captions is unclear.
Approach
Extend SugarCrepe with an LLM-generated second positive caption that is semantically equivalent but lexically different, forming image-caption triplets (P1, P2, N), with LLM generation and LLM validation followed by human correction, and score both image-to-text and text-only accuracy.
Key results
- Best VLM on SugarCrepe++ is FLAVA at 54.39% ITT on swap object versus 100% human; BLIP reaches 92.62% ITT on replace object.
- Models with better SugarCrepe scores do not always improve on SugarCrepe++: ALBEF and XVLM degrade relative to CLIP on SugarCrepe++ despite gains on SugarCrepe.
- Text-only (TOT) accuracy is much lower than image-text (ITT) accuracy for most VLMs, e.g., CLIP swap object TOT 19.74% versus ITT 45.18%, pointing to text encoders as the bottleneck.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune VLMs for image-text retrieval | ViLT and XVLM-16M · SugarCrepe++ | accuracy | Helped |
| Use multiple pre-training objectives rather than contrastive loss alone | VLMs · SugarCrepe++ | accuracy | Helped |
| Increase pre-training data size | CLIP variants · SugarCrepe++ | accuracy | Helped |
| Increase model size while keeping pre-training data at 400M | CLIP variants · SugarCrepe++ | accuracy | Mixed |
| Strengthen the text encoder with compositionality training | CLIP (NegCLIP and CLIP-SVLC) · SugarCrepe++ | accuracy | Helped |
| Use BLIP-SGVL compositionality training | VLMs · SugarCrepe++ | accuracy | Hurt |
| Use CyCLIP training | VLMs · SugarCrepe++ | accuracy | Hurt |
| Evaluate on SugarCrepe++ rather than SugarCrepe | ALBEF and XVLM · Replace Object | accuracy: 93.04 -> 76.27; 95.22 -> 79.42 | Hurt |
| Evaluate on SugarCrepe++ rather than SugarCrepe | BLIP · Replace Relation | accuracy: 68.35 -> 56.76 | Hurt |
| Add custom semantic-encoding instructions as the INSTRUCTOR prefix | Instructor-large · SugarCrepe++ TOT | accuracy: 59.42 -> 60.37 | Mixed |
Surprising findings
- Models that do better on the compositionality benchmark SugarCrepe (e.g., ALBEF, XVLM) degrade below CLIP on SugarCrepe++.
- Small sentence encoders such as BGE-small are competitive with 7B LLM encoders on replace relation (75.53 vs 80.51 range).
Limitations
- Second positive captions are generated by Mistral 7B and only corrected by two experts, so generation artifacts and bias may remain.
- Automatic validation assumes the image is captioned correctly and ignores the image when checking semantic equivalence.
Experimental setup
Datasets: SugarCrepe MS-COCO
Benchmarks: SugarCrepe++
Metrics: accuracy
Baselines: CLIP RoBERTa-ViT-B/32 ALIGN ALIP FLAVA ALBEF BLIP BLIP2 ViLT AltCLIP SegCLIP XVLM-4M
Builds on: SugarCrepe CREPE NegCLIP Rules Prompting
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.88)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.88)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.88)
- Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.