Research map / Vision-language and multimodal / Human and embodied modeling
Efficient Vision-Language Pre-training by Cluster Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cluster masking for CLIP-style contrastive vision-language pre-training, randomly masking clusters of visually similar patches (by RGB or embedding similarity) to improve representation quality and training speed on CC12M.
Problem
Masking image patches in contrastive vision-language pre-training speeds training but random masking can hurt representation quality, while semantic masking requires costly grouping mechanisms.
Approach
Randomly select anchor patches and mask all patches within a cosine-similarity threshold of them, using raw RGB (or RGB plus patch embedding) similarity, which yields random clustering each iteration without extra models or attention passes, providing a context-prediction signal alongside efficiency gains.
Key results
- Ours-Embedding averages 38.2% zero-shot classification vs 36.1% for CLIP at 0.64x normalized training time across 12 datasets.
- Linear probing: Ours-Embedding reaches 89.0 CIFAR-10 and 69.7 CIFAR-100 vs CLIP's 88.0 and 67.4.
- Ours-RGB0.3 reaches 36.8% average zero-shot vs FLIP's 33.5% at comparable cost, with 36.6% vs 34.4% on ImageNet-1K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use cluster masking instead of CLIP training without masking | Vision-language pre-training on CC12M · ImageNet-1K zero-shot classification | accuracy: +2.1% | Helped |
| use cluster masking instead of random patch masking | Vision-language pre-training on CC12M · ImageNet-1K zero-shot classification | accuracy: +5.5% | Helped |
| combine RGB features with patch embedding features for clustering rather than use RGB alone | Cluster masking · MS-COCO image-to-text retrieval | Recall@1: 22.97 -> 23.77 | Helped |
| increase anchor patch ratio from 0.01 to 0.03 (ablation) | Ours-RGB cluster masking · ImageNet-1K zero-shot classification | accuracy: 36.0 -> 36.6 | Helped |
| increase anchor patch ratio from 0.03 to 0.05 (ablation) | Ours-RGB cluster masking · ImageNet-1K zero-shot classification | accuracy: 36.6 -> 36.3 | Hurt |
| set minimum mask ratio β to 30% instead of 50% (ablation) | Ours-RGB · ImageNet-1K zero-shot classification | accuracy: 36.0 -> 36.6 | Helped |
| apply pixel normalization when computing the similarity matrix (ablation) | Ours cluster masking · ImageNet-1K zero-shot classification | accuracy: 35.5 -> 36.6 | Helped |
| use polynomial coefficient k=1 rather than k=0.5 for RGB/embedding interpolation (ablation) | Ours-Embedding · ImageNet-1K zero-shot classification | accuracy: 36.1 -> 36.3 | Helped |
| use polynomial coefficient k=2 rather than k=1 for RGB/embedding interpolation (ablation) | Ours-Embedding · ImageNet-1K zero-shot classification | accuracy: 36.3 -> 35.9 | Hurt |
| use cluster masking rather than CLIP without masking | Vision-language pre-training on CC12M · Flickr30k text-to-image retrieval | Recall@1: 58.50 -> 57.60 | Hurt |
| use cluster masking rather than CLIP without masking | Vision-language pre-training on CC12M · MS-COCO image-to-text retrieval | Recall@1: 23.89 -> 23.77 | Hurt |
Surprising findings
- Masking clusters of similar patches (cluster masking) beats random masking on ImageNet-1K even though fewer patches are seen, which the authors attribute to denoising by removing uninformative backgrounds.
Limitations
- Uses a uniform distance threshold for all images rather than per-image thresholds.
- Experiments limited to ViT-B/16 and CC12M pre-training data.
- Possible bag-of-words tendency since masked clusters can remove relational context; composition gains are mixed on Relation tests.
Experimental setup
Datasets: CC12M MS-COCO Flickr8k Flickr30k ImageNet CIFAR-10 CIFAR-100 SUGARCREPE
Benchmarks: MS-COCO Flickr8k Flickr30k ImageNet ImageNet variants CIFAR-10 CIFAR-100 Caltech101 Flowers Pets SUGARCREPE
Metrics: Recall@1 Recall@5 Recall@10 zero-shot accuracy linear probing accuracy normalized training time
Baselines: CLIP FLIP FLIPAttn K-Means masking
Builds on: CLIP FLIP ACLIP SemMAE MAE
Closest papers by meaning
- Understanding and Enhancing Mask-Based Pretraining towards Universal Representations (NeurIPS 2025 · similarity 0.88)
- SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining (CVPR 2024 · similarity 0.87)
- FFF: Fixing Flawed Foundations in Contrastive Pre-Training Results in Very Strong Vision-Language Models (CVPR 2024 · similarity 0.87)
- Improving fine-grained understanding in image-text pre-training (ICML 2024 · similarity 0.86)
- Osprey: Pixel Understanding with Visual Instruction Tuning (CVPR 2024 · similarity 0.86)
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.