Research map / Vision-language and multimodal / Human and embodied modeling
BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
BiMAC pretrains a CoCa-based vision-language model with contrastive learning, caption generation, and text-guided patch reconstruction with text-region grounding, reporting gains over CoCa on retrieval, captioning, and classification.
Problem
Vision-language alignment relies on global representations and unidirectional image-to-text flow (as in CoCa), limiting accurate reconstruction of visual content from text and fine-grained text-region correspondence.
Approach
Adds a text-to-image patch reconstruction decoder guided by text-region bipartite alignment to CoCa's contrastive and captioning objectives, creating bidirectional image-text interaction without masking.
Key results
- Zero-shot Flickr30K image-to-text R@1 of 46.8 vs 43.7 for CoCa; COCO 5K text-to-image R@1 17.5 vs 16.1.
- COCO captioning CIDEr 75.9 vs 71.8 for CoCa after fine-tuning.
- Zero-shot ImageNet-1K accuracy 31.6 vs 28.2 for CoCa; eight-dataset zero-shot average 58.4 vs 55.2 for CoCa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CoCa with BiMAC | CoCa-Base · Flickr30K 1K test and COCO 5K test | IR@1 / TR@1: 43.7 -> 46.8; 32.4 -> 36.8; 16.9 -> 19.4; 16.1 -> 17.5 | Helped |
| Replace CoCa with BiMAC | CoCa-Base · COCO captioning and NoCaps | CIDEr: 71.8 -> 75.9; 54.1 -> 55.3 | Helped |
| Replace CoCa with BiMAC | CoCa-Base · ImageNet-1K zero-shot classification | accuracy: 28.2 -> 31.6 | Helped |
| Replace CoCa with BiMAC | CoCa-Base · SPEC | I2T and T2I: 34.3 -> 39.7; 32.5 -> 36.2 | Helped |
| Replace CoCa with BiMAC | CoCa-Base · VALSE and Winoground | T / I / G: 69.5% to 74.1%; 11.2 to 14.9; 8.7 to 9.8 | Helped |
| Add caption generation and its reward function to MCL (ablation) | BiMAC · Flickr30K/MSCOCO/ImageNet-1K/CIFAR10/Caltech | Avg.: 47.9 -> 51.3 | Helped |
| Add GIT without grounding (ablation) | BiMAC · Flickr30K/MSCOCO/ImageNet-1K/CIFAR10/Caltech | Avg.: 51.3 -> 52.4 | Helped |
| Add grounding to GIT (ablation) | BiMAC · Flickr30K/MSCOCO/ImageNet-1K/CIFAR10/Caltech | Avg.: 52.4 -> 52.5 | Helped |
| Vary λGIT from 0.1 to 0.7 (ablation) | BiMAC · Flickr30K, MSCOCO, ImageNet, CIFAR10, Caltech | retrieval and classification: 69.1; 59.1; 37.9; 35.8; 30.5; 65.7; 67.4 -> 69.4; 59.5; 38.4; 36.9; 31.6; 68.5; 67.5 | Mixed |
| Increase λGIT to 1.5 (ablation) | BiMAC · Flickr30K, MSCOCO, ImageNet, CIFAR10, Caltech | retrieval and classification: 69.4; 59.5; 38.4; 36.9; 31.6; 68.5; 67.5 -> 68.7; 58.2; 37.3; 35.4; 30.7; 67.1; 66.5 | Hurt |
Surprising findings
- Reported lambda_GIT sensitivity is negligible across [0.1, 1.0], and GIT without grounding lowers performance relative to the full model.
Limitations
- Gains over CoCa are reported mainly against CoCa itself; comparisons to some baselines use numbers from other papers with differing settings
- Reconstruction-from-text objective and grounding details are sparsely specified (e.g., patch decoder inputs, negative sampling)
Experimental setup
Datasets: CC3M CC12M COCO Captions Flickr30K MSCOCO NoCaps ImageNet-1K DTD Flowers Food-101 CIFAR10 CIFAR100
Benchmarks: Flickr30K COCO NoCaps ImageNet-1K DTD Flowers Food-101 CIFAR10 CIFAR100 Caltech SUN397 SPEC
Metrics: Recall@1/5/10 BLEU@4 METEOR CIDEr SPICE zero-shot accuracy mTR mIR
Baselines: CoCa SyCoCa SPARC BEiT-v3 CoBIT ALIGN FILIP Florence BLIP StableRep VisMin-CLIP ViLT
Builds on: CoCa CLIP SyCoCa
Closest papers by meaning
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.90)
- SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment (ICML 2024 · similarity 0.90)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.89)
- CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.89)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.89)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.