Research map / Vision-language and multimodal / CLIP and zero-shot
Scaling Language-Image Pre-Training via Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 50% image masking with a larger batch (ablation) | CLIP training on LAION-400M · ImageNet-1K val zero-shot | accuracy: 68.6 -> 69.6 | Helped |
| Use 75% image masking with a larger batch (ablation) | CLIP training on LAION-400M · ImageNet-1K val zero-shot | accuracy: 68.6 -> 68.2 | No clear effect |
| Increase batch size (ablation) | FLIP with image masking · ImageNet-1K val zero-shot | accuracy: 68.5 -> 69.6 -> 70.4 | Helped |
| Randomly mask 50% of text tokens (ablation) | FLIP with 75% image masking · ImageNet-1K val zero-shot | accuracy: 68.2 -> 66.0 | Hurt |
| Prioritize masking text padding tokens (ablation) | FLIP with 75% image masking and 50% text masking · ImageNet-1K val zero-shot | accuracy: 66.0 -> 67.8 | Helped |
| Use intact images at inference instead of masked images (ablation) | FLIP · ImageNet-1K val zero-shot | accuracy: 66.4 -> 69.6; 60.9 -> 68.2 | Helped |
| Add unmasked tuning for 0.32 epoch (ablation) | FLIP · ImageNet-1K val zero-shot | accuracy: 69.6 -> 70.1; 68.2 -> 69.5 | Helped |
| Add MAE reconstruction loss (ablation) | FLIP · ImageNet-1K val zero-shot | accuracy: 69.6 -> 69.4; 68.2 -> 67.9 | Hurt |
| Scale the image and text encoders to Huge (ablation) | FLIP Large baseline · ImageNet-1K zero-shot | accuracy: 74.3 -> 75.5 | Helped |
| Scale pre-training data from 400M to 2B at fixed sampled data (ablation) | FLIP Large baseline · ImageNet-1K zero-shot | accuracy: 74.3 -> 75.8 | Helped |
| Train for 25.6B instead of 12.8B sampled data (ablation) | FLIP Large baseline · ImageNet-1K zero-shot | accuracy: 74.3 -> 73.9 | Hurt |
| Scale both model to Huge and data to 2B (ablation) | FLIP · ImageNet-1K zero-shot | accuracy: 74.3 -> 77.6 | Helped |
Closest papers by meaning
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.90)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.89)
- ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models (CVPR 2025 · similarity 0.88)
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.88)
- MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.