Research map / Vision-language and multimodal / Human and embodied modeling
Can the Parts Fool the Test? Counterfactual Pairing Cycles for Relational OOD
Yibo Zhou, Yirong Xiang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes counterfactual pairing cycles to build evaluations for relational out-of-distribution errors where the caption's relation is wrong for the image, controlling for shortcuts like caption style and object frequency.
Problem
Relational out-of-distribution errors (captions describing the wrong relation for an image) are hard to evaluate cleanly, because many existing negatives can be solved via superficial shortcuts such as caption style, template, length, or object frequency rather than checking the image-caption pairin
Approach
Uses counterfactual pairing cycles to construct evaluation pairs in which negatives cannot be distinguished from positives by surface cues, so that solving the task requires verifying the relational pairing. Details beyond the abstract are not available.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with cycle contrast loss on exact cycles instead of shortcut-contaminated BCE | rank-64 bilinear score on frozen CLIP ViT-B/32 · COCO exact cycles (OE stress test) | CycleAcc: 0.8817 -> 0.9270 | Improved |
| train four-entry BCE on exact cycles instead of cycle contrast loss (ablation) | rank-64 bilinear score on frozen CLIP ViT-B/32 · COCO exact cycles (OE stress test) | CycleAcc: 0.9270 -> 0.8853 | Hurt |
| inject marker prefix shortcut into OE negatives and validate on it | OE model · COCO exact cycles | CycleAcc: shortcut val AUROC 0.9998; CycleAcc 0.8817 vs 0.9270 | Mixed |
| increase bilinear rank from 1 to 64 (ablation) | bilinear probe on frozen CLIP · COCO object-matched cycles | CycleAcc: 0.784 -> 0.959 | Improved |
| increase bilinear rank from 1 to 64 (ablation) | bilinear probe on frozen CLIP · COCO strict present left/right cycles | CycleAcc: 0.514 -> 0.505 | No clear effect |
| increase bilinear rank from 1 to 64 (ablation) | bilinear probe on frozen CLIP · COCO strict same-pair relation cycles | CycleAcc: 0.567 -> 0.599 | Improved |
| increase bilinear rank from 1 to 64 (ablation) | bilinear probe on frozen CLIP · COCO strict size cycles | CycleAcc: 0.687 -> 0.787 | Improved |
| increase rank from 8 to 16 in synthetic rank control (ablation) | interaction probe · Synthetic rank bottleneck control | AUROC: 0.7658 -> 0.7575 | No clear effect |
| use additive image-only, text-only or additive probes under exact cycles | additive probes · COCO and GQA exact cycles | CycleAcc: 0.500 ± 0.000 | No clear effect |
| inject marginal artifact into unblocked synthetic diagnostic | marginal probe · Synthetic controls | AUROC: 0.5005 -> 1.0000 | Improved |
| use pair-aware features instead of bag features (ablation) | binding representation control · Synthetic binding control | AUROC: 0.5000 -> 0.9919 | Improved |
Surprising findings
- Many existing relational OOD evaluations can be solved without checking the image-caption pairing, due to confounds in caption style, template, length, and object frequency.
Closest papers by meaning
- SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis (NeurIPS 2026 · similarity 0.86)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.85)
- WikiDO: A New Benchmark Evaluating Cross-Modal Retrieval for Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox (NeurIPS 2024 · similarity 0.85)
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.84)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
