Research map / Vision-language and multimodal / CLIP and zero-shot
SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SyCoCa, which extends CoCa with a text-guided masked image modeling head and attentive masking to add bidirectional local image-text interaction, pretrained from scratch on CC12M (main) and CC3M (ablations).
Problem
CoCa's image captioning head provides only unidirectional image-to-text local interaction, limiting fine-grained multimodal alignment between images and text.
Approach
Adds a text-guided masked image modeling (TG-MIM) head that reconstructs masked image patches conditioned on text via cross-attention, and uses attentive masking to select text-relevant patches (most relevant for IC, least relevant for TG-MIM) based on token-wise max similarity, yielding bidirectional local interaction.
Key results
- On CC3M ablation, SyCoCa averages 35.9 vs CoCa 31.9 across retrieval and classification (+12.5%).
- Zero-shot Flickr30K text-to-image R@1 improves from 31.3 (CoCa) to 35.9 in the CC12M setting, and VQA test-dev improves from 39.1 to 42.6.
- Zero-shot classification on 8 coarse-grained datasets averages 42.2 vs CoCa 41.9 (SyCoCa wins 7 of 8); linear probing on 7 fine-grained datasets averages 65.6 vs 64.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TG-MIM and attentive masking to CoCa's ITC + IC objectives (ablation) | CoCa · CC3M zero-shot retrieval and classification | Avg.: 31.9 -> 35.9 (+12.5\%) | Helped |
| Add attentive masking to TG-MIM (ablation) | CoCa with ITC + IC + TG-MIM · CC3M zero-shot retrieval and classification | Avg.: 32.5 -> 35.9 | Helped |
| Use random masking instead of attentive masking (ablation) | CoCa with ITC + IC + TG-MIM · CC3M zero-shot retrieval and classification | Avg.: 35.9 -> 32.4 | Hurt |
| Use MIM instead of text-guided MIM (ablation) | CoCa with ITC + IC and attentive masking · CC3M zero-shot retrieval and classification | Avg.: 35.9 -> 32.5 | Hurt |
| Set attentive masking ratios to r_l 25% and r_h 75% rather than 50% each (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.9 -> 35.0 | Hurt |
| Set attentive masking ratios to r_l 75% and r_h 25% rather than 50% each (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.9 -> 34.7 | Hurt |
| Vary r_l while holding r_h at 50% (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.5, 35.9, 35.6 | No clear effect |
| Vary r_h while holding r_l at 50% (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.4, 35.9, 36.1 | No clear effect |
| Vary TG-MIM loss weight from 0.1 to 1.0 (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.2, 35.1, 35.9 | No clear effect |
| Set TG-MIM loss weight to 2.0 rather than 1.0 (ablation) | SyCoCa · CC3M zero-shot retrieval and classification | Avg.: 35.9 -> 34.8 | Hurt |
| Add bidirectional local interaction with TG-MIM and attentive masking | CoCa · Flickr30K image-to-text retrieval | R@1: 44.2 -> 46.6 | Helped |
| Add bidirectional local interaction with TG-MIM and attentive masking | CoCa · Flickr30K text-to-image retrieval | R@1: 31.3 -> 35.9 | Helped |
Surprising findings
- TG-MIM alone without attentive masking has little impact on performance, and random masking slightly hurts TG-MIM relative to MIM.
Limitations
- Main experiments use only CC12M and CC3M, far smaller than CoCa's pretraining data, so comparisons with large-scale models are not like-for-like.
- Ablations are run on CC3M and show mixed results on some classification datasets (e.g., Pets and C-10 drop in some settings).
Experimental setup
Datasets: CC12M CC3M Flickr30K MSCOCO NoCaps VQA ImageNet CIFAR-10 STL-10 Caltech101 LAION-2B COYO-700M
Benchmarks: Flickr30K MSCOCO NoCaps VQA ImageNet-1K ImageNet-V2 ImageNet-A ImageNet-R CIFAR-10 CIFAR-100 STL-10 Caltech101
Metrics: R@1 R@5 R@10 mTR mIR BLEU@4 METEOR CIDEr SPICE VQA accuracy
Baselines: CoCa CLIP ALIGN FILIP Florence BEiT-3 EVA-CLIP SigLIP CLIPA-v2 OpenCLIP
Builds on: CoCa CLIP MAE SimMIM
Closest papers by meaning
- BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning (AAAI 2025 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.89)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.89)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.