Research map / Vision-language and multimodal / CLIP and zero-shot
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how English-only filtering of web image-text pretraining data affects cultural and socioeconomic diversity in SigLIP models, and proposes few-shot geo-localization as a cultural diversity metric.
Problem
Contrastive VLMs are evaluated mainly on Western-centric benchmarks (ImageNet, COCO), and English-only data filtering common in CLIP/ALIGN/SigLIP training may hurt cultural and socioeconomic diversity.
Approach
Trains SigLIP models on English-only, global multilingual, and machine-translated global WebLI subsets under a compute-matched regime, showing that global pretraining followed by brief English fine-tuning or data mixing improves cultural diversity without sacrificing standard benchmarks, and introduces few-shot geo-localization as a metric.
Key results
- Global (tl) pretraining vs English-only: GLDv2 zero-shot 49.46% vs 43.84% (+5.62 points) while ImageNet zero-shot drops 70.36% to 68.23% for tl.
- Few-shot geo-localization on GeoDE (country, 25 shots): en 23.24% vs tl 36.54%, a +13.30 point gap.
- Fine-tuning tl-pretrained models on English data for about 50k steps keeps ImageNet accuracy while improving GLDv2 0-shot accuracy above 45.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace English-only pretraining with global multilingual pretraining | SigLIP · Dollar Street, GLDv2, GeoDE, and MaRVL Concepts | zero-shot accuracy: 48.52 $\pm 0.53\%$ -> 49.96 $\pm 0.71\%$; 43.84 $\pm 0.52 \%$ -> 49.46 $\pm 1.17 \%$; 91.82 $\pm 0.39 \%$ -> 92.84 $\pm | Helped |
| Replace English-only pretraining with global multilingual pretraining | SigLIP · ImageNet | zero-shot accuracy: 70.36 $\pm 0.28 \%$ -> 66.81 $\pm 0.18 \%$ | Hurt |
| Replace English-only pretraining with global multilingual pretraining | SigLIP · COCO image-to-text and text-to-image | Recall@1: 59.28 $\pm 0.90 \%$ -> 55.81 $\pm 0.59 \%$; 42.91 $\pm 0.56 \%$ -> 38.09 $\pm 0.58 \%$ | Hurt |
| Translate non-English captions to English during global-data pretraining | SigLIP · Dollar Street, GLDv2, GeoDE, and MaRVL Concepts | zero-shot accuracy: +1.44\%; +5.62\%; +1.02\%; +1.66\% | Helped |
| Use global translated data instead of English-only data | SigLIP · XM3600 English captions retrieval | Recall@1: -0.86\%; -3.26\% | No clear effect |
| Use global translated data instead of English-only data | SigLIP · few-shot geo-localization on Dollar Street, XM3600, and GeoDE | geo-localization accuracy: +2.97\%; +7.42\%; +9.82\%; +9.06\% | Helped |
| Pretrain on global translated data, then fine-tune on English-only data | SigLIP · GLDv2 zero-shot classification and ImageNet zero-shot classification | zero-shot accuracy: 44\% to over 45.5% | Mixed |
| Use global translated pretraining followed by English-only fine-tuning instead of English-only pretraining followed by global fine-tuning | SigLIP · culturally diverse benchmarks | zero-shot accuracy | Helped |
| Mix English-only and translated global training data rather than fine-tuning between them | SigLIP · GLDv2 and ImageNet | zero-shot accuracy: 46.43% vs. 46.24%; 70.28% vs. 68.45% | Mixed |
| Apply quality filtering to the training data | SigLIP English-only and translated-data models · Dollar Street, GLDv2, GeoDE, ImageNet, and COCO | zero-shot accuracy / Recall@1: 46.05 % -> 48.28 %; 28.21 % -> 30.67 %; 90.53 % -> 90.57 %; 66.96 % -> 66.32 %; 56.72 % -> 52.80 % | Mixed |
Surprising findings
- XM3600 retrieval shows no statistically significant differences between en, globe and tl models, so multilingual benchmarks fail to capture cultural diversity.
- English-only models perform better on ImageNet and COCO, yet worse on culturally diverse benchmarks, so popular benchmarks are at odds with cultural diversity.
Limitations
- Only contrastive encoder-only SigLIP models studied; generative VLMs not evaluated.
- Culture is operationalized as country, region, or income group performance; no precise definition of culture.
- Relationship between cultural diversity and social bias is not addressed.
Experimental setup
Datasets: WebLI Dollar Street GeoDE GLDv2 XM3600 MaRVL ImageNet COCO
Benchmarks: Dollar Street GLDv2 GeoDE MaRVL XM3600 ImageNet COCO
Metrics: zero-shot accuracy top-1 recall few-shot geo-localization accuracy
Baselines: SigLIP English-only (en) SigLIP global multilingual (globe) SigLIP translated (tl)
Builds on: SigLIP CLIP WebLI Big Vision
Closest papers by meaning
- See It from My Perspective: How Language Affects Cultural Bias in Image Understanding (ICLR 2025 · similarity 0.92)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.92)
- World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models (CVPR 2026 · similarity 0.90)
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.89)
- Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping (AAAI 2026 · similarity 0.89)
- Beyond Aesthetics: Cultural Competence in Text-to-Image Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.