Research map / Vision-language and multimodal / CLIP and zero-shot
CoBIT: A Contrastive Bi-directional Image-Text Generation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoBIT, a unicoder-decoder model jointly pre-trained with contrastive, image-to-text and text-to-image generation losses on web image-text data, with image/text unicoders that switch between encoding and decoding modes.
Problem
Vision-language foundation models are typically pre-trained with only one of three objectives (contrastive, image-to-text generation, text-to-image generation), even though all three use the same image-text pair data and could complement each other.
Approach
A unicoder-decoder architecture where image and text Transformer unicoders share parameters across encoding (bidirectional) and decoding (causal) modes, feeding a cross-attention decoder, so that contrastive, I2T and T2I losses are optimized jointly in one framework.
Key results
- CoBIT-Large achieves 82.7% zero-shot ImageNet classification accuracy and 9.37 zero-shot MS-COCO FID.
- CoBIT-Large reaches 44.8 zero-shot MS-COCO captioning CIDEr, surpassing SimVLM by 12.6.
- After fine-tuning, CoBIT-Large gets 86.44% ImageNet linear probing and 4.62 FID on MS-COCO text-to-image generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add I2T and T2I generation objectives to contrastive loss (ablation) | CoBIT · ImageNet zero-shot | accuracy: 70.8 -> 71.1 | Helped |
| add T2I loss to I2T loss (ablation) | CoBIT · VQA | score: 68 -> 65.4 | Hurt |
| add I2T loss to T2I loss (ablation) | CoBIT · MS-COCO zero-shot text-to-image generation | FID: 12.6 -> 13.2 | Hurt |
| add contrastive loss to the two generation losses (ablation) | CoBIT · VQA | score: 65.4 -> 66.9 | Helped |
| add contrastive loss to the two generation losses (ablation) | CoBIT · MS-COCO zero-shot text-to-image generation | FID: 13.2 -> 13.3 | No clear effect |
| replace both image and text encoders with unicoders (ablation) | CoBIT · VQA | score: 65.9 -> 66.9 | Helped |
| replace both image and text encoders with unicoders (ablation) | CoBIT · MS-COCO zero-shot captioning | CIDEr: 32.9 -> 37.9 | Helped |
| replace both image and text encoders with unicoders (ablation) | CoBIT · MS-COCO zero-shot text-to-image generation | FID: 13.8 -> 13.31 | Helped |
| initialize the text unicoder from CoCa instead of training from scratch (ablation) | CoBIT · ImageNet zero-shot | accuracy: 75.02 -> 75.35 | Helped |
| initialize the text unicoder from CoCa instead of training from scratch (ablation) | CoBIT · VQA | score: 68.55 -> 68.48 | No clear effect |
| initialize the text unicoder from CoCa instead of training from scratch (ablation) | CoBIT · MS-COCO zero-shot text-to-image generation | FID: 11.63 -> 11.42 | Helped |
| increase I2T loss weight from 0.1 to 1 while T2I weight is 1 (ablation) | CoBIT · MS-COCO zero-shot text-to-image generation | FID: 12.6 -> 16.33 | Hurt |
Surprising findings
- The image-to-text and text-to-image generation losses partially contradict each other: adding T2I loss drops VQA accuracy by 2.6, and adding I2T loss raises zero-shot FID by 0.6.
Limitations
- T2I and I2T generation objectives contradict each other somewhat, per the ablations.
- Zero-shot ImageNet remains about 2% below CoCa-Large, attributed to smaller batch size (30k vs 64k).
- Failure cases include misreading compositional prompts, size attributes, and rendering text in images.
Experimental setup
Datasets: ALIGN JFT-4B WebLI ImageNet MS-COCO Flickr30k VQA v2 SNLI-VE
Benchmarks: ImageNet Flickr30k MS-COCO VQA v2 SNLI-VE PartiPrompts
Metrics: top-1 accuracy recall@1 recall@5 CIDEr FID VQA accuracy
Baselines: CLIP ALIGN FILIP Florence CoCa SimVLM VLKD Parti LDM Flamingo DALL-E 2 OFA
Builds on: CoCa Parti CLIP ALIGN ViT-VQGAN
Closest papers by meaning
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.88)
- Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.87)
- BiMAC: Bidirectional Multimodal Alignment in Contrastive Learning (AAAI 2025 · similarity 0.87)
- Discriminative Probing and Tuning for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
- Harmonizing Visual Text Comprehension and Generation (NeurIPS 2024 · similarity 0.87)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.