Research map / Vision-language and multimodal / CLIP and zero-shot
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add masked self-distillation to CLIP | CLIP · ImageNet-1K zero-shot | accuracy: 37.6 -> 44.5 | Helped |
| Add MLM loss to MaskCLIP (ablation) | MaskCLIP · ImageNet-1K zero-shot | accuracy: 42.8 -> 44.5 | Helped |
| Add MLM loss to MaskCLIP (ablation) | MaskCLIP · ImageNet-1K finetuning | accuracy: 83.6 -> 83.6 | No clear effect |
| Add masked self-distillation loss (ablation) | MaskCLIP · ImageNet-1K finetuning | accuracy: 82.4 -> 83.6 | Helped |
| Use cross-entropy with an online tokenizer instead of MSE (ablation) | MaskCLIP distillation · ImageNet-1K zero-shot | accuracy: 43.8 -> 44.5 | Helped |
| Use cross-entropy with an online tokenizer instead of MSE (ablation) | MaskCLIP distillation · ImageNet-1K finetuning | accuracy: 83.6 -> 83.6 | No clear effect |
| Use a 1-layer visual decoder instead of a 2-layer decoder (ablation) | MaskCLIP · ImageNet-1K zero-shot | accuracy: 43.7 -> 44.5 | Helped |
| Use a 1-layer visual decoder instead of a 4-layer decoder (ablation) | MaskCLIP · ImageNet-1K linear probing | accuracy: 72.5 -> 73.7 | Helped |
| Set distillation loss weight to 0.05 instead of 1 (ablation) | MaskCLIP · ImageNet-1K zero-shot | accuracy: 38.5 -> 44.5 | Helped |
| Set MLM loss weight to 0.05 instead of 1 (ablation) | MaskCLIP · ImageNet-1K zero-shot | accuracy: 36.5 -> 44.5 | Helped |
| Add a text decoder instead of using no text decoder (ablation) | MaskCLIP · ImageNet-1K zero-shot | accuracy: 43.5 -> 44.5 | Helped |
| Use a 4-layer rather than an 8-layer text decoder (ablation) | MaskCLIP · Flickr30K image-to-text retrieval | R@1: 67.5 -> 70.1 | Helped |
Closest papers by meaning
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.90)
- Expediting Contrastive Language-Image Pretraining via Self-Distilled Encoders (AAAI 2024 · similarity 0.89)
- Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners (ICLR 2024 · similarity 0.89)
- Efficient Vision-Language Pre-training by Cluster Masking (CVPR 2024 · similarity 0.89)
- MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization (ICML 2024 · similarity 0.87)
- FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.