Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
MoDE: CLIP Data Experts via Clustering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoDE, which clusters image-caption data hierarchically via captions, trains a CLIP data expert per coarse cluster, and ensembles experts at inference by routing task metadata to fine-grained cluster centers.
Problem
Web-crawled image-caption pairs contain noisy captions, so in-batch contrastive learning produces false negatives that hurt CLIP training quality and efficiency.
Approach
Cluster training captions into disjoint data subsets so each expert sees semantically coherent pairs with fewer false negatives, train experts asynchronously from a shared partially trained seed, and route zero-shot tasks to experts via similarity between class names and fine-grained cluster centers, without parameter updates at inference.
Key results
- MoDE-4 with four ViT-B/16 experts on 2.5B pairs improves CLIP benchmark average over MetaCLIP by +3.7% and outperforms OpenCLIP ViT-L/14 by 1.5% while using less than 35% of OpenAI CLIP ViT-L/14 GPU-hours.
- On COCO with ViT-B/16, MoDE-4 gains +3.3% R@1 for image-to-text and +2.7% R@1 for text-to-image retrieval over OpenCLIP/MetaCLIP baselines.
- Naive ensembling of two models trained without clustering (Full-2) gives no gain over the baseline (58.3 vs 58.2 average on 400M ViT-B/32).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use four clustered data experts instead of the MetaCLIP baseline | MoDE · CLIP benchmark (2.5B image-caption pairs, ViT-B/16) | average accuracy: 63.5 -> 67.2 | Helped |
| Use four clustered data experts instead of the MetaCLIP baseline | MoDE · COCO text retrieval (2.5B image-caption pairs, ViT-B/16) | R@1: 59.4 -> 62.7 | Helped |
| Use four clustered data experts instead of the MetaCLIP baseline | MoDE · COCO image retrieval (2.5B image-caption pairs, ViT-B/16) | R@1: 41.4 -> 44.1 | Helped |
| Randomly split the training data into two subsets instead of clustering (ablation) | MoDE-2 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 58.6 -> 57.7 | Hurt |
| Randomly split the training data into two subsets instead of clustering (ablation) | MoDE-2 · CLIP benchmark (2.5B image-caption pairs, ViT-B/32) | average accuracy: 59.8 -> 60.0 | No clear effect |
| Train two models on the same unclustered training data (ablation) | MoDE-2 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 58.2 -> 58.3 | No clear effect |
| Train two models on the same unclustered training data (ablation) | MoDE-2 · CLIP benchmark (2.5B image-caption pairs, ViT-B/32) | average accuracy: 59.8 -> 60.0 | No clear effect |
| Use one-step clustering instead of two-step clustering (ablation) | MoDE-2 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 58.5 -> 58.0 | Hurt |
| Use coarse-grained cluster centers rather than fine-grained centers for ensemble weights (ablation) | MoDE-2 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 58.6 -> 58.5 | Hurt |
| Use coarse-grained cluster centers rather than fine-grained centers for ensemble weights (ablation) | MoDE-4 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 59.0 -> 58.7 | Hurt |
| Use DINOv2 image embeddings instead of SimCSE language embeddings for clustering (ablation) | MoDE-2 · CLIP benchmark (400M image-caption pairs, ViT-B/32) | average accuracy: 58.6 -> 58.1 | Hurt |
| Use CLIP seed image embeddings instead of SimCSE language embeddings for clustering (ablation) | MoDE-2 · ImageNet (400M image-caption pairs, ViT-B/32) | accuracy: 66.1 -> 64.7 | Hurt |
Surprising findings
- Random data splits give a drop on 400M pairs (57.7 vs 58.2) despite ensembling, showing clustering is essential.
- Image-embedding-based clustering underperforms MetaCLIP on ImageNet, while SimCSE text embeddings work best for clustering.
Limitations
- Experts start from a shared seed model partially trained on all data, so they are not fully independent.
- Coarse-level clustering forces disjoint groups of fine-grained clusters, which the authors note may be limiting.
- Evaluated mainly on zero-shot classification and retrieval; not yet extended to generative models.
Experimental setup
Datasets: MetaCLIP 400M image-caption pairs MetaCLIP 2.5B image-caption pairs CLIP benchmark datasets COCO Flickr30k ImageNet variants
Benchmarks: CLIP benchmark (27 datasets) ImageNet-Sketch ImageNet-V2 ImageNet-A ImageNet-O ImageNet-R COCO retrieval Flickr30k retrieval
Metrics: zero-shot accuracy average accuracy R@1 R@5 R@10 linear probe accuracy
Baselines: OpenAI CLIP OpenCLIP MetaCLIP CoarseCluster OneStep-2
Builds on: CLIP MetaCLIP SimCSE balanced K-means deep mixture of experts
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.87)
- CLIP-KD: An Empirical Study of CLIP Model Distillation (CVPR 2024 · similarity 0.86)
- CLIPLoss and Norm-Based Data Selection Methods for Multimodal Contrastive Learning (NeurIPS 2024 · similarity 0.86)
- Meta CLIP 2: A Worldwide Scaling Recipe (NeurIPS 2025 · similarity 0.86)
- CLIP the Bias: How Useful is Balancing Data in Multimodal Learning? (ICLR 2024 · similarity 0.86)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.