Research map / Vision-language and multimodal / Human and embodied modeling
COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs
Tiep Le, Vasudev Lal, Phillip Howard
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Augment real MS-COCO training data with COCO-Counterfactuals at a 60% share | CLIP · Flickr30k 1K test | image-text retrieval mean: 89.06_{0.1} -> 89.53_{0.2} | Improved |
| Augment real MS-COCO training data with COCO-Counterfactuals at a 50% share | CLIP · Flickr30k 1K test | image-text retrieval mean: 89.06_{0.1} -> 89.05_{0.1} | No clear effect |
| Increase the counterfactual share from 50% to 60% (ablation) | CLIP trained on MS-COCO and COCO-Counterfactuals · Flickr30k 1K test | image-text retrieval mean: 89.05_{0.1} -> 89.53_{0.2} | Improved |
| Increase the counterfactual share from 60% to 67% (ablation) | CLIP trained on MS-COCO and COCO-Counterfactuals · Flickr30k 1K test | image-text retrieval mean: 89.53_{0.2} -> 89.41_{0.1} | Hurt |
| Augment real MS-COCO training data with COCO-Counterfactuals at a 50% share | CLIP · zero-shot image recognition across six datasets | mean accuracy: 78.85_{0.2} -> 79.46_{0.2} | Improved |
| Increase the counterfactual share from 50% to 60% (ablation) | CLIP trained on MS-COCO and COCO-Counterfactuals · zero-shot image recognition across six datasets | mean accuracy: 79.46_{0.2} -> 79.20_{0.2} | Hurt |
| Train on COCO-Counterfactuals rather than an equivalent amount of real MS-COCO data | CLIP · withheld COCO-Counterfactuals test | image-text retrieval mean: 77.03 -> 83.09 | Improved |
| Remove human-annotator-incorrect synthetic pairs from training (ablation) | CLIP trained with MS-COCO and COCO-Counterfactuals · Flickr30k test | image-text retrieval mean | No clear effect |
| Evaluate on minimally edited COCO-Counterfactuals instead of MS-COCO | BridgeTower · zero-shot image-text retrieval | text retrieval R@1: -51% | Hurt |
| Evaluate on minimally edited COCO-Counterfactuals instead of MS-COCO | Flava · zero-shot image-text retrieval | text retrieval R@1: -57% | Hurt |
| Evaluate on COCO-Counterfactuals rather than MS-COCO | CLIP · zero-shot image-text retrieval | text retrieval R@1: -21% | Hurt |
| Evaluate on COCO-Counterfactuals rather than MS-COCO | CLIP · zero-shot image-text retrieval | image retrieval R@1: +5% | Improved |
Closest papers by meaning
- Improving Vision and Language Concepts Understanding with Multimodal Counterfactual Samples (ECCV 2024 · similarity 0.90)
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits (NeurIPS 2025 · similarity 0.86)
- Revisiting Counterfactual Problems in Referring Expression Comprehension (CVPR 2024 · similarity 0.86)
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning (AAAI 2024 · similarity 0.85)
- Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion (ICML 2026 · similarity 0.85)
- Retrieving Counterfactuals Improves Visual In-Context Learning (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
