Research map / Vision-language and multimodal / Human and embodied modeling
Multilingual Diversity Improves Vision-Language Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Translates the 128M-sample DataComp medium web crawl into English with NLLB, re-filters with DFN/CLIP, and shows that including translated non-English captions improves English vision benchmarks (ImageNet, shifts, retrieval, DataComp average) and GeoDE.
Problem
Data curation for vision-language pretraining filters web data toward English, discarding useful non-English image-text pairs and likely hurting performance on English-centric vision benchmarks and geographically diverse tasks.
Approach
Fix the language medium by translating all captions of a raw web crawl to English with NLLB, re-run image-text filtering on translated captions, and combine translated and raw-caption subsets so that multilingual diversity enters the training set while still improving English vision tasks.
Key results
- Combining filtered raw and filtered translated captions (51.2M) at 1.28B steps gives ImageNet 0.456 vs 0.414 for filtered raw captions (+4.2 points) and 2.1-point average gain over 38 tasks.
- On GeoDE, training on translated captions improves worst-region (Africa) accuracy by 5.5% over filtered raw captions.
- Filtering on translated captions makes non-English-origin samples the majority of the top-20% subset, and MAUVE between English and translated non-English captions is only 0.449.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| filter using translated multilingual captions instead of raw captions | CLIP ViT-B/32 · DataComp (128M steps) | ImageNet accuracy: 0.316 -> 0.329 | Helped |
| filter using translated multilingual captions instead of raw captions | CLIP ViT-B/32 · DataComp (128M steps) | ImageNet shifts accuracy: 0.260 -> 0.275 | Helped |
| filter using translated multilingual captions instead of raw captions | CLIP ViT-B/32 · DataComp (128M steps) | Retrieval: 0.282 -> 0.296 | Helped |
| filter using translated multilingual captions instead of raw captions | CLIP ViT-B/32 · GeoDE worst-region | accuracy: 0.688 -> 0.709 | Helped |
| filter using translated multilingual captions instead of raw captions | CLIP ViT-B/32 · DataComp 38 tasks | average accuracy: 0.350 -> 0.359 | Helped |
| combine raw and translated filtered subsets while deduplicating shared images | CLIP ViT-B/32 · DataComp 38 tasks | average accuracy: 0.350 -> 0.364 | Helped |
| replace raw captions with English translations without re-filtering (ablation) | Filtered raw captions · DataComp (128M steps) | DataComp 38-task average accuracy: 0.350 -> 0.331 | Hurt |
| replace translated captions with BLIP2 synthetic captions while keeping selected images fixed (ablation) | Filtered translated captions, top 20% · DataComp (128M steps) | DataComp 38-task average accuracy: 0.359 -> 0.336 | Hurt |
| replace translated captions with BLIP2 synthetic captions while keeping selected images fixed (ablation) | Filtered translated captions, top 30% · DataComp (128M steps) | DataComp 38-task average accuracy: 0.351 -> 0.341 | Hurt |
| combine raw and translated filtered subsets, retaining duplicate images with both captions | CLIP ViT-B/32 · DataComp (1.28B steps) | DataComp 38-task average accuracy: 0.414 -> 0.435 | Helped |
| use translated multilingual captions instead of raw captions | CLIP ViT-B/32 · DataComp (1.28B steps) | DataComp 38-task average accuracy: 0.414 -> 0.414 | No clear effect |
| use translated multilingual captions instead of raw captions | CLIP ViT-B/32 · GeoDE regions | accuracy: 5.5% | Helped |
Surprising findings
- Replacing raw captions with their English translations without re-filtering lowers performance across the board, so the gain requires re-filtering after translation.
- Filtered translated captions outperform filtered raw captions on ImageNet and retrieval despite ImageNet's Western bias.
Limitations
- Best baseline is not state-of-the-art on DataComp; comparisons with other filtering methods (HypeSampler, T-MARS) not explored.
- Translation introduces artifacts and loses richness; backtranslation quality is low for some languages.
- Filtering is restricted to CLIP-style cosine similarity scores.
Experimental setup
Datasets: DataComp medium pool (128M) GeoDE ImageNet ImageNet shifts Flickr30K MSCOCO WinoGAViL Dollar Street
Benchmarks: ImageNet ImageNet shifts (V2, Sketch, A, O, R, ObjectNet) Flickr30K MSCOCO WinoGAViL GeoDE DataComp 38 tasks Dollar Street
Metrics: zero-shot accuracy retrieval score worst-region accuracy average accuracy over 38 tasks MAUVE score
Baselines: Filtered raw captions Filtered English-only captions OpenAI CLIP filtering HypeSampler LAION-Translated
Builds on: DataComp NLLB Data Filtering Networks CLIP LAION-Translated
Closest papers by meaning
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.93)
- No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models (NeurIPS 2024 · similarity 0.92)
- See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.90)
- AltDiffusion: A Multilingual Text-to-Image Diffusion Model (AAAI 2024 · similarity 0.90)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.90)
- Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.