Research map / Vision-language and multimodal / CLIP and zero-shot
Meta CLIP 2: A Worldwide Scaling Recipe
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta CLIP 2 trains CLIP from scratch on worldwide (native, multilingual) web image-text pairs using scaled multilingual metadata, per-language curation with language-specific thresholds, scaled seen pairs, and ViT-H/14 capacity to break the curse of multilinguality.
Problem
CLIP is trained on English-only curated data, discarding most non-English web data; prior multilingual CLIP efforts rely on distillation, translation, or private data, and multilingual training degrades English performance (the curse of multilinguality).
Approach
Extend Meta CLIP's transparent, model-free curation to worldwide data by building per-language metadata (300+ languages), per-language substring matching and balancing with language-specific thresholds derived from a shared tail-proportion, scaling seen pairs with the global batch size (2.3x), and using ViT-H/14 capacity so English and non-English data mutually benefit.
Key results
- Meta CLIP 2 ViT-H/14 with worldwide 29B seen pairs reaches 81.3% ImageNet zero-shot, up from 80.4% for its English-only counterpart at 13B pairs.
- It sets new multilingual SoTA among compared models, e.g., XM3600 64.3% (I→T avg recall@1 side), Babel-IN 50.2%, and CVQA 57.4%.
- Meta CLIP 2 surpasses mSigLIP and SigLIP 2 on several multilingual benchmarks (e.g., Babel-IN +3.8%, Flickr30k-200 +7.7%/+7%) with fewer seen pairs (72% of SigLIP series).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale seen training pairs from 1.0× to 2.3× for worldwide data (ablation) | Meta CLIP 2 ViT-H/14 · ImageNet val | zero-shot accuracy: 79.5 -> 81.3 | Helped |
| scale seen training pairs from 1.0× to 2.3× for worldwide data (ablation) | Meta CLIP 2 ViT-H/14 · Babel-ImageNet | zero-shot accuracy: 47.1 -> 50.2 | Helped |
| use ViT-H/14 instead of ViT-L/14 for worldwide training with 2.3× seen pairs (ablation) | Meta CLIP 2 · ImageNet val | zero-shot accuracy: 78.8 -> 81.3 | Helped |
| train on worldwide rather than English-only data (ablation) | Meta CLIP 2 ViT-H/14 (1.0×) · ImageNet val | zero-shot accuracy: 80.4 -> 79.5 | Hurt |
| train on worldwide rather than English-only data (ablation) | Meta CLIP 2 ViT-H/14 (2.3×) · ImageNet val | zero-shot accuracy: 80.5 -> 81.3 | Helped |
| remove English filtering while keeping English metadata (ablation) | ViT-B/32 CLIP · ImageNet val | zero-shot accuracy: 67.5 -> 66.9 | Hurt |
| isolate language-specific metadata and matching rather than merge all languages (ablation) | ViT-B/32 CLIP · ImageNet val | zero-shot accuracy: 62.1 -> 61.1 | Helped |
| use a language-specific threshold rather than a shared English threshold (ablation) | ViT-B/32 CLIP with language isolation · ImageNet val | zero-shot accuracy: 61.1 -> 64.7 | Helped |
| use a language-specific threshold rather than a shared English threshold (ablation) | ViT-B/32 CLIP with language isolation · XM3600 image-to-text | recall@1: 49.4 -> 50.0 | Helped |
| use XLM-V instead of mT5 tokenizer (ablation) | ViT-B/32 Meta CLIP 2 Worldwide (1.0×) · Babel-ImageNet | zero-shot accuracy: 31.5 -> 32.7 | Helped |
| use XLM-V instead of Gemma tokenizer (ablation) | ViT-B/32 Meta CLIP 2 Worldwide (1.0×) · XM3600 text-to-image | recall@1: 36.1 -> 40.0 | Helped |
| train on worldwide rather than English-only data (ablation) | Meta CLIP 2 ViT-H/14 (13B seen pairs) · GLDv2 | zero-shot accuracy: 52.8 -> 65.8 | Helped |
Surprising findings
- Training on worldwide data with a ViT-L/14 model still shows the curse of multilinguality, while ViT-H/14 breaks it and improves English ImageNet accuracy.
- Removing the English-only filter alone slightly lowers ImageNet accuracy (67.5% to 66.9%) on ViT-B/32.
Limitations
- Worldwide benchmarks remain sparse and biased toward NA+EU and English, making evaluation of worldwide coverage difficult.
- Non-English-only training at 1.0x seen pairs underperforms, so the scaling of seen pairs is required, increasing compute.
- Some metadata tokenizers for scriptio continua languages rely on community-developed tools.
Experimental setup
Datasets: worldwide web image-text pairs (Common Crawl-based) Wikipedia dumps (May 2024) multilingual WordNet Wikipedia page titles
Benchmarks: ImageNet val SLIP 26 DataComp 37 Babel-ImageNet XM3600 CVQA Flickr30k-200 XTD-10 XTD-200 Dollar Street GeoDE GLDv2
Metrics: zero-shot accuracy recall@1 few-shot geo-localization accuracy alignment uniformity
Baselines: OpenAI CLIP (LAION-5B) mSigLIP SigLIP 2 Meta CLIP DFN OpenCLIP
Builds on: CLIP Meta CLIP OpenAI CLIP XLM-V tokenizer
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.93)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.90)
- Embracing Language Inclusivity and Diversity in CLIP through Continual Language Learning (AAAI 2024 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.