Research map / Vision-language and multimodal / Human and embodied modeling
Object-centric binding in Contrastive Language-Image Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OC-CLIP, which adds a slot-based binding module and a structured scene-graph similarity score to CLIP-like models to improve multi-object compositional understanding without extra hard negatives.
Problem
CLIP-style contrastive models struggle to bind attributes to objects and to understand spatial relations in multi-object scenes, and hard-negative augmentation approaches do not explicitly address this binding problem.
Approach
Integrates object-centric inductive biases into pretrained CLIP: an inverted cross-attention binding module lets scene-graph node embeddings (from an LLM parser) attend to image patches to form competing visual slots, and a structured similarity score matches nodes to slots and models relations as text-conditioned constraints, trained with a local graph contrastive loss.
Key results
- OC-CLIP improves SugarCrepe swap-attribute by +16.5% over in-domain finetuned OpenCLIP and reaches over 89% on COCO-spatial and 92% on GQA-spatial, up from random chance.
- Trained from scratch on CC3M+CC12M, OC-CLIP gains +12.8% zero-shot ImageNet accuracy over CLIP (reported as 45.0 vs 32.2 on 15M).
- On the synthetic PUG setting, OC-CLIP reaches 97% on seen object pairs with no hard negatives, while CLIP with 70% hard negatives plateaus at 81%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OC-CLIP instead of in-domain OpenCLIP-FT | OpenCLIP · SugarCrepe swap-attribute | accuracy: +16.5% | Helped |
| Use OC-CLIP instead of in-domain OpenCLIP-FT | OpenCLIP · SugarCrepe swap-object | accuracy: +20.4% | Helped |
| Use OC-CLIP instead of in-domain OpenCLIP-FT | OpenCLIP · SugarCrepe replace-relation | accuracy: +4.1% | Helped |
| Use OC-CLIP instead of OpenCLIP-FT | OpenCLIP · COCO-spatial | accuracy: +44.1% | Helped |
| Use OC-CLIP instead of OpenCLIP-FT | OpenCLIP · GQA-spatial | accuracy: +43.6% | Helped |
| Use OC-CLIP instead of OpenCLIP-FT | OpenCLIP · ARO-Relation | accuracy: +34.8% | Helped |
| Use OC-CLIP with no hard negatives instead of OpenCLIP trained with 70% hard negatives | OpenCLIP · PUG seen object pairs | accuracy: 81% to 97% | Helped |
| Train OC-CLIP from scratch on CC3M+CC12M instead of CLIP | ViT-B-16 · ImageNet zero-shot | accuracy: +12.8% | Helped |
| Add competitive cross-attention (ablation) | OC-CLIP · SugarCrepe attribute binding | accuracy: 85.9 to 89.0 | Helped |
| Add local graph contrastive loss (ablation) | OC-CLIP · SugarCrepe relational | accuracy: 72.8 to 80.5 | Helped |
| Add attention layers (ablation) | OC-CLIP · SugarCrepe relational | accuracy: 77.6 to 80.5 | Helped |
| Increase default tokens from 1 to 4 (ablation) | OC-CLIP · SugarCrepe attribute binding | accuracy: 89.0 to 87.6 | Hurt |
Surprising findings
- Adding more hard-negative augmentation to OpenCLIP plateaus and does not improve swap-attribute binding even at large scale (DAC).
- The relation scoring component is sensitive to the scene-graph parser choice: spaCy and T5 parsers reduce relation-replacement accuracy relative to an LLM parser.
Limitations
- Relies on an external LLM-based text parser to extract scene graphs, and parsing quality affects performance.
- Binding module and scoring are trained from scratch, so the learned space may only generalize within the training vocabulary.
- Added computational overhead of about 2.2x FLOPs at the base ViT size from the binding module and graph-based text encoding.
Experimental setup
Datasets: COCO Visual Genome GQA CC3M CC12M PUG
Benchmarks: SugarCrepe ARO COCO-spatial GQA-spatial What's Up ImageNet Winoground ELEVATER VL-Checklist
Metrics: accuracy
Baselines: OpenCLIP-FT BLIP XVLM NegCLIP CE-CLIP CC-CLIP CLIP-SVLC DAC-LLM DAC-SAM
Builds on: CLIP Slot Attention NegCLIP OpenCLIP
Closest papers by meaning
- Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models (NeurIPS 2026 · similarity 0.92)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- CLIP Under the Microscope: A Fine-Grained Analysis of Multi-Object Representation (CVPR 2025 · similarity 0.90)
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding (CVPR 2024 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.