Research map / Vision-language and multimodal / Human and embodied modeling
BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BiVLC, a bidirectional vision-language compositionality benchmark extending SugarCrepe with synthetic hard negative images, and trains CLIP models on new hard-negative text and image datasets (TROHN-Text, TROHN-Img) that improve compositionality.
Problem
Vision-language compositionality benchmarks evaluate only image-to-text retrieval, leaving text-to-image retrieval unmeasured; models that look strong in one direction may be weak in the other.
Approach
Add a generated hard-negative image for each SugarCrepe hard-negative caption, yielding two images and two captions per instance and thus both I2T and T2I retrieval examples, filtered by human annotators; also train CLIP with hard-negative synthetic images to improve both directions.
Key results
- CLIP_TROHN-Img reaches 69.25 group score on BiVLC versus 60.96 for the best prior contrastive model CLIP_COCO, though VQAScore-XXL reaches 76.47 and humans 86.80.
- Human group score on BiVLC is 86.80 versus 76.47 for the best model, with models consistently weaker on text-to-image retrieval than image-to-text.
- CLIP_TROHN-Text achieves 93.40 on SugarCrepe, the best contrastive result, but only 57.48 group score on BiVLC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train CLIP with hard negative images using TROHN-Img instead of COCO training data | CLIP · BiVLC | Group: 60.96 -> 69.25 | Helped |
| Train CLIP with hard negative images using TROHN-Img instead of COCO training data | CLIP · BiVLC | T2I: 63.89 -> 71.84 | Helped |
| Train CLIP on TROHN-Text hard negative captions instead of COCO training data | CLIP · BiVLC | Group: 60.96 -> 57.48 | Hurt |
| Train CLIP on TROHN-Text hard negative captions instead of COCO training data | CLIP · SugarCrepe | accuracy: 84.66 -> 93.40 | Helped |
| Train CLIP on TROHN-Img hard negative images instead of COCO training data | CLIP · SugarCrepe | accuracy: 84.66 -> 89.40 | Helped |
| Use text-to-image retrieval rather than image-to-text retrieval | multimodal models · BiVLC | accuracy: 82.75 -> 63.89 | Hurt |
| Evaluate bidirectional retrieval with BiVLC instead of image-to-text retrieval with SugarCrepe | multimodal models · model ranking | accuracy | Mixed |
| Use bidirectional retrieval with BiVLC rather than image-to-text retrieval with SugarCrepe | Open CapPa · model ranking | accuracy: 90.59 -> 41.97 | Hurt |
| Train CLIP with hard negative images | CLIP · BiVLC | T-pos2I: 72.79 -> 81.45 | Helped |
| Use synthetic-versus-natural detection as the retrieval strategy | CLIP-TROHN-Img/Det · BiVLC | Group: 69.25 -> 26.42 | Hurt |
| Evaluate models on the Swap category rather than Replace | multimodal models · BiVLC | Group | Hurt |
Surprising findings
- Strong SugarCrepe performance does not transfer to BiVLC: CLIP_TROHN-Text scores 93.40 on SugarCrepe but is worse than CLIP_COCO on BiVLC.
- Open CapPa is near-perfect on SugarCrepe Add categories but performs poorly on BiVLC Add (group 9.26).
Limitations
- About 61% of generated hard-negative training images are noisy or ambiguous, likely limiting CLIP_TROHN-Img.
- Captions only in English and only contrastive models were trained.
- Inter-annotator agreement for image selection is moderate (Cohen's Kappa 0.49).
Experimental setup
Datasets: SugarCrepe COCO 2017 Pick-a-Pic TROHN-Text TROHN-Img
Benchmarks: BiVLC SugarCrepe
Metrics: I2T accuracy T2I accuracy group score
Baselines: CLIP CLIP_COCO NegCLIP GNM Open CapPa VQAScore-XL VQAScore-XXL
Builds on: SugarCrepe Winoground NegCLIP Generative Negative Mining DPO
Closest papers by meaning
- Vision-by-Language for Training-Free Compositional Image Retrieval (ICLR 2024 · similarity 0.89)
- Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data (CVPR 2025 · similarity 0.89)
- CoLLM: A Large Language Model for Composed Image Retrieval (CVPR 2025 · similarity 0.88)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.88)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.