Research map / Vision-language and multimodal / CLIP and zero-shot
Three Towers: Flexible Contrastive Learning with Pretrained Image Models
Jannik Kossen, Mark Collier, Basil Mustafa, Xiao Wang, Xiaohua Zhai, Lucas Beyer, Andreas Peter Steiner, Jesse Berent, Rodolphe Jenatton, Effrosyni Kokiopoulou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add third tower with pretrained embeddings (3T) vs from-scratch baseline | CLIP/ALIGN · Flickr* img2txt retrieval, L, IN-21k | top-1 recall: 75.6 -> 80.0 | Improved |
| 3T instead of LiT, IN-21k | L-scale towers · Flickr* txt2img | top-1 recall: 49.3 -> 60.9 | Improved |
| 3T instead of LiT, JFT | L-scale towers · COCO img2txt | top-1 recall: 52.7 -> 54.4 | Improved |
| 3T instead of baseline, IN-21k | L-scale towers · average classification | accuracy: 68.4 -> 71.4 | Improved |
| LiT instead of baseline, IN-21k | L-scale towers · average classification | accuracy: 68.4 -> 68.3 | No clear effect |
| LiT instead of baseline on Stanford Cars zero-shot/few-shot | IN-21k · Cars 10-shot | accuracy: 81.5 -> 41.9 | Hurt |
| LiT instead of 3T on Birds | IN-21k · Birds 10-shot | accuracy: 65.0 -> 83.4 | Improved |
| mismatched pretrained scale (B/32 pretrained, L/16 tower) | LiT · zero-shot full average | accuracy: 66.4 -> 61.7 | Hurt |
| Places365 pretrained model | LiT · zero-shot full average | accuracy: 47.5 -> 29.4 | Hurt |
| remove image-text contrastive loss (ablation) | 3T · average of tasks | performance difference: -26.63 ± 10.61 | Hurt |
| remove image-tower transfer loss (ablation) | 3T · average of tasks | performance difference: -1.19 ± 0.75 | Hurt |
| remove text-tower transfer loss (ablation) | 3T · average of tasks | performance difference: -2.77 ± 0.91 | Hurt |
Closest papers by meaning
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.86)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.86)
- Retrieval-Enhanced Contrastive Vision-Text Models (ICLR 2024 · similarity 0.86)
- Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · similarity 0.86)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.86)
- $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
