Research map / Vision-language and multimodal / CLIP and zero-shot
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MobileCLIP, a family of mobile-efficient CLIP image-text encoders (hybrid CNN-transformer) trained with multi-modal reinforced training on a reinforced DataComp dataset (DataCompDR) that stores synthetic captions and ensemble CLIP teacher embeddings.
Problem
Large CLIP image-text encoders are too slow and memory-heavy for mobile deployment, and small-scale CLIP training is too costly to explore efficient architectures and too weak to yield accurate small models.
Approach
Extend offline dataset reinforcement to multimodal CLIP training: store augmentation parameters, CoCa synthetic captions, and embeddings from an ensemble of strong CLIP teachers once, then train students with a distillation affinity-matching loss plus standard CLIP loss at no extra training-time teacher cost; pair with efficient hybrid image (MCi) and text (MCt) encoders using structural reparamet
Key results
- MobileCLIP-S2 is 2.3x faster than the previous best ViT-B/16-based CLIP while more accurate (abstract).
- MobileCLIP-B achieves 65.2% average on 38 datasets, +2.9% over SigLIP-B/16 while being 26.3% smaller.
- Training on DataCompDR shows 10x-1000x learning efficiency vs non-reinforced training; MobileCLIP-B reaches 71.7% IN-val zero-shot with 12M subset after 120 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with DataCompDR-12M instead of DataComp-12M | ViT-B/16:Base · ImageNet-val, 0.37B seen samples | zero-shot accuracy: 50.1 -> 65.3 | Helped |
| Train with DataCompDR-12M instead of DataComp-12M | ViT-B/16:Base · ImageNet-val, 0.74B seen samples | zero-shot accuracy: 53.5 -> 69.1 | Helped |
| Train with DataCompDR-12M instead of DataComp-12M | ViT-B/16:Base · ImageNet-val, 1.48B seen samples | zero-shot accuracy: 55.7 -> 71.7 | Helped |
| Add synthetic captions alongside real captions during CLIP training (ablation) | ViT-B/16:Base on DataComp-12M · ImageNet-val | zero-shot accuracy: 44.5 -> 51.9 | Helped |
| Add synthetic captions alongside real captions during CLIP training (ablation) | ViT-B/16:Base on DataComp-12M · Flickr30k | recall@1: 41.8 -> 69.3 | Helped |
| Use strong image augmentations for distillation (ablation) | ViT-B/16:Base on DataComp-12M · ImageNet-val | zero-shot accuracy: +4.8% | Helped |
| Use strong image augmentations for distillation (ablation) | ViT-B/16:Base on DataComp-12M · Flickr30k | recall@1: +4.4% | Helped |
| Use an ensemble of strong CLIP teachers instead of no teacher (ablation) | ViT-B/16:Base on DataComp-12M · ImageNet-val | zero-shot accuracy: +2.4% | Helped |
| Set distillation loss coefficient λ to 1.0 instead of 0 (ablation) | ViT-B/16:Base on DataComp-12M · ImageNet-val | zero-shot accuracy: 44.5 -> 54.5 | Helped |
| Set distillation loss coefficient λ to 1.0 instead of 0 (ablation) | ViT-B/16:Base on DataComp-12M · Flickr30k | recall@1: 41.8 -> 66.1 | Helped |
| Use 2 self-attention layers instead of 0 in MCt (ablation) | MCt text encoder · ImageNet-val | zero-shot accuracy: 57.9 -> 60.2 | Helped |
| Increase synthetic captions per image from 0 to 1 (ablation) | ViT-B/16:Base on DataCompDR-12M · ImageNet-val | zero-shot accuracy: 60.67 -> 64.88 | Helped |
Surprising findings
- Most accurate models are not necessarily the best teachers (ensemble teacher analysis).
Limitations
- Reinforced dataset requires one-time expensive generation of synthetic captions and teacher embeddings with large models
- Storage overhead of DataCompDR (1.9 TB for 12.8M samples; 140 TB for 1.28B)
Experimental setup
Datasets: DataComp-1B DataComp-12M CC-12M YFCC-15M LAION-400M
Benchmarks: ImageNet val ImageNet-V2 ImageNet-A ImageNet-O ImageNet-R ObjectNet MSCOCO Flickr30k DataComp 38-dataset evaluation ARO
Metrics: zero-shot accuracy recall@1 average performance on 38 datasets latency (ms)
Baselines: CLIP ViT-B/16 (DataComp-12M) TinyCLIP SLIP CLIPA SigLIP-B/16 DFN-B/16 DataComp-B/32-256 OpenAI CLIP
Builds on: CLIP dataset reinforcement FastViT DataComp TinyCLIP
Closest papers by meaning
- Demystifying CLIP Data (ICLR 2024 · similarity 0.89)
- SuperCLIP: CLIP with Simple Classification Supervision (NeurIPS 2025 · similarity 0.88)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.87)
- un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP (NeurIPS 2025 · similarity 0.87)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.87)
- Modeling Caption Diversity in Contrastive Vision-Language Pretraining (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.